Queue Support¶
The queue support module makes it possible to add log records to a queue
system. This is useful for distributed setups where you want multiple
processes to log to the same backend. Currently supported are ZeroMQ as
well as the multiprocessing
Queue
class.
ZeroMQ¶
-
class
logbook.queues.
ZeroMQHandler
(uri=None, level=0, filter=None, bubble=False, context=None)[source]¶ A handler that acts as a ZeroMQ publisher, which publishes each record as json dump. Requires the pyzmq library.
The queue will be filled with JSON exported log records. To receive such log records from a queue you can use the
ZeroMQSubscriber
.Example setup:
handler = ZeroMQHandler('tcp://127.0.0.1:5000')
-
context
= None¶ the zero mq context
-
socket
= None¶ the zero mq socket.
-
-
class
logbook.queues.
ZeroMQSubscriber
(uri=None, context=None)[source]¶ A helper that acts as ZeroMQ subscriber and will dispatch received log records to the active handler setup. There are multiple ways to use this class.
It can be used to receive log records from a queue:
subscriber = ZeroMQSubscriber('tcp://127.0.0.1:5000') record = subscriber.recv()
But it can also be used to receive and dispatch these in one go:
with target_handler: subscriber = ZeroMQSubscriber('tcp://127.0.0.1:5000') subscriber.dispatch_forever()
This will take all the log records from that queue and dispatch them over to target_handler. If you want you can also do that in the background:
subscriber = ZeroMQSubscriber('tcp://127.0.0.1:5000') controller = subscriber.dispatch_in_background(target_handler)
The controller returned can be used to shut down the background thread:
controller.stop()
-
context
= None¶ the zero mq context
-
dispatch_forever
()¶ Starts a loop that dispatches log records forever.
-
dispatch_in_background
(setup=None)¶ Starts a new daemonized thread that dispatches in the background. An optional handler setup can be provided that pushed to the new thread (can be any
logbook.base.StackedObject
).Returns a
ThreadController
object for shutting down the background thread. The background thread will already be running when this function returns.
-
dispatch_once
(timeout=None)¶ Receives one record from the socket, loads it and dispatches it. Returns True if something was dispatched or False if it timed out.
-
recv
(timeout=None)[source]¶ Receives a single record from the socket. Timeout of 0 means nonblocking, None means blocking and otherwise it’s a timeout in seconds after which the function just returns with None.
-
socket
= None¶ the zero mq socket.
-
MultiProcessing¶
-
class
logbook.queues.
MultiProcessingHandler
(queue, level=0, filter=None, bubble=False)[source]¶ Implements a handler that dispatches over a queue to a different process. It is connected to a subscriber with a
multiprocessing.Queue
:from multiprocessing import Queue from logbook.queues import MultiProcessingHandler queue = Queue(-1) handler = MultiProcessingHandler(queue)
-
class
logbook.queues.
MultiProcessingSubscriber
(queue=None)[source]¶ Receives log records from the given multiprocessing queue and dispatches them to the active handler setup. Make sure to use the same queue for both handler and subscriber. Idaelly the queue is set up with maximum size (
-1
):from multiprocessing import Queue queue = Queue(-1)
It can be used to receive log records from a queue:
subscriber = MultiProcessingSubscriber(queue) record = subscriber.recv()
But it can also be used to receive and dispatch these in one go:
with target_handler: subscriber = MultiProcessingSubscriber(queue) subscriber.dispatch_forever()
This will take all the log records from that queue and dispatch them over to target_handler. If you want you can also do that in the background:
subscriber = MultiProcessingSubscriber(queue) controller = subscriber.dispatch_in_background(target_handler)
The controller returned can be used to shut down the background thread:
controller.stop()
If no queue is provided the subscriber will create one. This one can the be used by handlers:
subscriber = MultiProcessingSubscriber() handler = MultiProcessingHandler(subscriber.queue)
-
dispatch_forever
()¶ Starts a loop that dispatches log records forever.
-
dispatch_in_background
(setup=None)¶ Starts a new daemonized thread that dispatches in the background. An optional handler setup can be provided that pushed to the new thread (can be any
logbook.base.StackedObject
).Returns a
ThreadController
object for shutting down the background thread. The background thread will already be running when this function returns.
-
dispatch_once
(timeout=None)¶ Receives one record from the socket, loads it and dispatches it. Returns True if something was dispatched or False if it timed out.
-
Other¶
-
class
logbook.queues.
ThreadedWrapperHandler
(handler)[source]¶ This handled uses a single background thread to dispatch log records to a specific other handler using an internal queue. The idea is that if you are using a handler that requires some time to hand off the log records (such as the mail handler) and would block your request, you can let Logbook do that in a background thread.
The threaded wrapper handler will automatically adopt the methods and properties of the wrapped handler. All the values will be reflected:
>>> twh = ThreadedWrapperHandler(TestHandler()) >>> from logbook import WARNING >>> twh.level_name = 'WARNING' >>> twh.handler.level_name 'WARNING'
-
class
logbook.queues.
SubscriberGroup
(subscribers=None, queue_limit=10)[source]¶ This is a subscriber which represents a group of subscribers.
This is helpful if you are writing a server-like application which has “slaves”. This way a user is easily able to view every log record which happened somewhere in the entire system without having to check every single slave:
subscribers = SubscriberGroup([ MultiProcessingSubscriber(queue), ZeroMQSubscriber('tcp://localhost:5000') ]) with target_handler: subscribers.dispatch_forever()
Base Interface¶
-
class
logbook.queues.
SubscriberBase
[source]¶ Baseclass for all subscribers.
-
dispatch_in_background
(setup=None)[source]¶ Starts a new daemonized thread that dispatches in the background. An optional handler setup can be provided that pushed to the new thread (can be any
logbook.base.StackedObject
).Returns a
ThreadController
object for shutting down the background thread. The background thread will already be running when this function returns.
-
-
class
logbook.queues.
ThreadController
(subscriber, setup=None)[source]¶ A helper class used by queue subscribers to control the background thread. This is usually created and started in one go by
dispatch_in_background()
or a comparable function.
-
class
logbook.queues.
TWHThreadController
(wrapper_handler)[source]¶ A very basic thread controller that pulls things in from a queue and sends it to a handler. Both queue and handler are taken from the passed
ThreadedWrapperHandler
.