Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,34 @@ Or alternatively spin up a `rtlog::LogProcessingThread`
```c++
rtlog::LogProcessingThread thread(logger, PrintMessage, std::chrono::milliseconds(10));
```

## Customizing the queue type

If you don't want to use the SPSC moodycamel queue, you can provide your own queue type.

** IT IS UP TO YOU TO ENSURE THE QUEUE YOU PROVIDE IS LOCK-FREE AND REAL-TIME SAFE **

The queue must have the following:
```c++
template <typename T>
class MyQueue
{
public:
using value_type = T;

MyQueue(int capacity);
bool try_dequeue(T& item); // MUST return false if the queue is empty

bool try_enqueue(T&& item);
// OR
bool try_enqueue(const T& item);
};
```

Then, when creating the logger, provide the queue type as a template parameter:

```c++
using RealtimeLogger = rtlog::Logger<ExampleLogData, MAX_NUM_LOG_MESSAGES, MAX_LOG_MESSAGE_LENGTH, gSequenceNumber, MyQueue>;
```

You can see an example of wrapping a known rt-safe queue in `examples/custom_queue_example`.
1 change: 1 addition & 0 deletions examples/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
add_subdirectory(everlog)
add_subdirectory(custom_queue_example)
29 changes: 29 additions & 0 deletions examples/custom_queue_example/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
if (NOT TARGET farbot)
include(FetchContent)

FetchContent_Declare(farbot
GIT_REPOSITORY https://github.com/hogliux/farbot
GIT_TAG 0416705394720c12f0d02e55c144e4f69bb06912
)
# Note we do not "MakeAvailable" here, because farbot does not fully work via FetchContent
if(NOT farbot_POPULATED)
FetchContent_Populate(farbot)
endif()
add_library(farbot INTERFACE)
add_library(farbot::farbot ALIAS farbot)

target_include_directories(farbot INTERFACE
$<BUILD_INTERFACE:${farbot_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>
)
endif()

add_executable(custom_queue_example
customqueuemain.cpp
)

target_link_libraries(custom_queue_example
PRIVATE
rtlog::rtlog
farbot::farbot
)
47 changes: 47 additions & 0 deletions examples/custom_queue_example/customqueuemain.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#include <farbot/fifo.hpp>
#include <rtlog/rtlog.h>

template <typename T> class FarbotMPSCQueueWrapper {
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@atsushieno I incorporated your work here as an example for future readers. Would you please review this?

I'll make sure you get marked as a co-author in the git commit, as I took much of this verbatim from your branch :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes everything looks much greater with the customizible queues! I'm glad my changes were of some help.

I don't try hard to push my MPSC stuff in the main code from "example use case", but what I'm concerned however is that people would like to have default implementation for MPSC anyway, and they would simply ask you to incorporate that farbot version (or anything equivalent) after all, just like there is the original SPSC implementation.
For my own purpose I will keep my friendly fork version (maybe not even a fork, just an addon header would work), and I'm a bit concerned that it may be more widely referenced than your original repo as people would usually want to use the logging functionality in multi-threaded apps.
As a library author in this world I understand that I wouldn't like to incorporate external deps that might not match my taste (or being asked to do so), but having third-party fork more popular (especially widely used without my fixes) would not be a great experience.
IF that were the case in the future, I'm always happy to share my stuff and feel free to take them back into this repo.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense.

How about this: I submit this change, then either myself or you (if you had time and wanted to) "swap" farbot and readerwriterqueue. Farbot becomes the main dependency, we add two versions of the wrapper - one single producer and one multi-producer. readerwriterqueue becomes the example dependency.

Overall, my only concern with incorporating farbot AND readerwriterqueue together is that we would be downloading more dependencies to user's machines. I already feel partly bad downloading stb, libfmt AND readerwriterqueue. Most libraries I know and trust don't download anything and are "self reliant". I would like to not add additional dependencies to the mix if possible. I agree that it is reasonable for users to want both single producer and multi producer versions of rtlog.

If we swapped in farbot as the primary queue, this would keep the dependency count the same and add more flexibility for the user. It would also be easy for anyone to add a new queue on their own if they wanted it.

What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be the best solution I agree :-)
I'm working on some other stuff now, but when I have time and if it's not done then I'll take the task (I don't mind overlaps, wouldn't waste too much time anyway).

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I'll ping you in any reviews if I take on the work, please do the same for me!

farbot::fifo<T,
farbot::fifo_options::concurrency::single, // Consumer
farbot::fifo_options::concurrency::multiple, // Producer
farbot::fifo_options::full_empty_failure_mode::
return_false_on_full_or_empty, // consumer_failure_mode
farbot::fifo_options::full_empty_failure_mode::
overwrite_or_return_default> // producer_failure_mode

mQueue;

public:
using value_type = T;

FarbotMPSCQueueWrapper(int capacity) : mQueue(capacity) {}

bool try_enqueue(T &&item) { return mQueue.push(std::move(item)); }
bool try_dequeue(T &item) { return mQueue.pop(item); }
};

struct LogData {};

std::atomic<size_t> gSequenceNumber{0};

int main() {
rtlog::Logger<LogData, 128, 128, gSequenceNumber, FarbotMPSCQueueWrapper>
logger;
logger.Log({}, "Hello, World!");

logger.PrintAndClearLogQueue(
[](const LogData &data, size_t sequenceNumber, const char *fstring, ...) {
(void)data;
std::array<char, 128> buffer;

va_list args;
va_start(args, fstring);
vsnprintf(buffer.data(), buffer.size(), fstring, args);
va_end(args);

printf("{%zu} %s\n", sequenceNumber, buffer.data());
});

return 0;
}
118 changes: 107 additions & 11 deletions include/rtlog/rtlog.h
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
#include <cstdarg>
#include <cstdio>
#include <thread>
#include <type_traits>
#include <utility>

#if !defined(RTLOG_USE_FMTLIB) && !defined(RTLOG_USE_STB)
// The default behavior to match legacy behavior is to use STB
Expand Down Expand Up @@ -54,16 +56,89 @@ enum class Status {
Error_MessageTruncated = 2,
};

namespace detail {

template <typename LogData, size_t MaxMessageLength> struct BasicLogData {
LogData mLogData{};
size_t mSequenceNumber{};
std::array<char, MaxMessageLength> mMessage{};
};

template <typename T, typename = void>
struct has_try_enqueue_by_move : std::false_type {};

template <typename T>
struct has_try_enqueue_by_move<
T, std::void_t<decltype(std::declval<T>().try_enqueue(
std::declval<typename T::value_type &&>()))>> : std::true_type {};

template <typename T>
inline constexpr bool has_try_enqueue_by_move_v =
has_try_enqueue_by_move<T>::value;

template <typename T, typename = void>
struct has_try_enqueue_by_value : std::false_type {};

template <typename T>
struct has_try_enqueue_by_value<
T, std::void_t<decltype(std::declval<T>().try_enqueue(
std::declval<typename T::value_type const &>()))>> : std::true_type {
};

template <typename T>
inline constexpr bool has_try_enqueue_by_value_v =
has_try_enqueue_by_value<T>::value;

template <typename T>
inline constexpr bool has_try_enqueue_v =
has_try_enqueue_by_move_v<T> || has_try_enqueue_by_value_v<T>;

template <typename T, typename = void>
struct has_try_dequeue : std::false_type {};

template <typename T>
struct has_try_dequeue<T, std::void_t<decltype(std::declval<T>().try_dequeue(
std::declval<typename T::value_type &>()))>>
: std::true_type {};

template <typename T>
inline constexpr bool has_try_dequeue_v = has_try_dequeue<T>::value;

template <typename T, typename = void>
struct has_value_type : std::false_type {};

template <typename T>
struct has_value_type<T, std::void_t<typename T::value_type>> : std::true_type {
};

template <typename T>
inline constexpr bool has_value_type_v = has_value_type<T>::value;

template <typename T, typename = void>
struct has_int_constructor : std::false_type {};

template <typename T>
struct has_int_constructor<T, std::void_t<decltype(T(std::declval<int>()))>>
: std::true_type {};

template <typename T>
inline constexpr bool has_int_constructor_v = has_int_constructor<T>::value;
} // namespace detail

// On earlier versions of compilers (especially clang) you cannot
// rely on defaulted template template parameters working as intended
// This overload explicitly has 1 template paramter which is what
// `Logger` expects, it uses the default 512 from ReaderWriterQueue as
// the hardcoded MaxBlockSize
template <typename T> using rtlog_SPSC = moodycamel::ReaderWriterQueue<T, 512>;

/**
* @brief A logger class for logging messages.
* This class allows you to log messages of type LogData.
* This type is user defined, and is often the additional data outside the
* format string you want to log. For instance: The log level, the log region,
* the file name, the line number, etc. See examples or tests for some ideas.
*
* TODO: Currently is built on a single input/single output queue. Do not call
* Log or PrintAndClearLogQueue from multiple threads.
*
* @tparam LogData The type of the data to be logged.
* @tparam MaxNumMessages The maximum number of messages that can be enqueud at
* once. If this number is exceeded, the logger will return an error.
Expand All @@ -72,11 +147,38 @@ enum class Status {
* @tparam SequenceNumber This number is incremented when the message is
* enqueued. It is assumed that your non-realtime logger increments and logs it
* on Log.
* @tparam QType is the configurable underlying queue. By default it is a SPSC
* queue from moodycamel. WARNING! It is up to the user to ensure this queue
* type is real-time safe!!
*
* Requirements on QType:
* 1. Is real-time safe
* 2. Accepts one type template paramter for the type to be queued
* 3. Has a constructor that takes an integer which will be the queue's
* capacity
* 4. Has methods `bool try_enqueue(T &&item)` and/or `bool
* try_enqueue(const T &item)` and `bool try_dequeue(T &item)`
*/
template <typename LogData, size_t MaxNumMessages, size_t MaxMessageLength,
std::atomic<std::size_t> &SequenceNumber>
std::atomic<std::size_t> &SequenceNumber,
template <typename> class QType = rtlog_SPSC>
class Logger {
public:
using InternalLogData = detail::BasicLogData<LogData, MaxMessageLength>;
using InternalQType = QType<InternalLogData>;

static_assert(
detail::has_int_constructor_v<InternalQType>,
"QType must have a constructor that takes an int - `QType(int)`");
static_assert(detail::has_value_type_v<InternalQType>,
"QType must have a value_type - `using value_type = T;`");
static_assert(detail::has_try_enqueue_v<InternalQType>,
"QType must have a try_enqueue method - `bool try_enqueue(T "
"&&item)` and/or `bool try_enqueue(const T &item)`");
static_assert(
detail::has_try_dequeue_v<InternalQType>,
"QType must have a try_dequeue method - `bool try_dequeue(T &item)`");

/*
* @brief Logs a message with the given format and input data.
*
Expand Down Expand Up @@ -261,13 +363,7 @@ class Logger {
}

private:
struct InternalLogData {
LogData mLogData{};
size_t mSequenceNumber{};
std::array<char, MaxMessageLength> mMessage{};
};

moodycamel::ReaderWriterQueue<InternalLogData> mQueue{MaxNumMessages};
InternalQType mQueue{MaxNumMessages};
};

/**
Expand Down
Loading