Tags: jaalcorn/logstash-input-file
Tags
Fix: sincedb_clean_after not being respected (logstash-plugins#276) In certain cases, in read mode sincedb is not updated. There was no periodic check for sincedb updates that would cause the sincedb_clean_after entries to cleanup. The cleanup relied on new files being discovered or new content being added to existing files -> causing sincedb updates. The fix here is to periodically flush sincedb (from the watch loop). Besides, to make the process more deterministic, there's a minor change to make sure the same "updated" timestamp is used to mark the last changed time. resolves logstash-plugins#250 expected to also resolve logstash-plugins#260
Fix: skip sincedb eviction if read mode completion deletes file durin… …g flush If the value of the setting `sincedb_clean_after` is too short (1 ms) it happens that during the content reading loop (in read mode) the loops itself trigger the cleanup of the same value from the SincedbCollection and then it also try to access it again creating an error. This commit resolves the problem guarding against the presence in the sincebd_collection before accessing it. Fixes: logstash-plugins#272 Closes: logstash-plugins#273
Fix: watched files performance with huge filesets (logstash-plugins#268) Plugin tracks all discovered files in a map-like collection (`FileWatch::WatchedFilesCollection`). The collection design was to re-sort all elements every time the collection gets updated ... this obviously does not scale with 10000s of watched files. Starting up LS with 60k files in a watched directory, locally, takes **\~ an hour before these changes**. **After these changes** for the plugin to start processing files it takes **~2 seconds**. Since the collection is known to have a noticeable (memory) footprint, there's changes towards reducing intermediate garbage - which also contributed to the decision to move the collection to native. Notable changes to tracking watched files with `FileWatch::WatchedFilesCollection`: - predictable `log(size)` modification (and access) times - implementation has clear locking semantics, previously operations weren't 100% atomic - hopefully, cleaner API as the collection re-resembles a *WatchedFile -> String (path)* map - the collection now needs an explicit update whenever a file changes (on re-stat), previous implicitness of picking up changes might have been the reasoning behind re-sorts (could be decoupled further by making `WatchedFile` immutable but there's other state anyway) --- Apart from the necessary changes to resolve the performance bottleneck, there's also a quick review of (trace) logging - annoying to not only see partial traces even in debug/trace levels. Few unused methods and instance variables were removed for a better code reading experience.
Fix: release watched files on completion (in read-mode) (logstash-plu… …gins#271) not doing so leads to a steady increase in watched collection's size over time (esp. in use-cases where user is pulling in new files). the left-over file is never to be processed again - it's being deleted anyway using the completion handler.
[DOC] Doc improvements (logstash-plugins#266) * Doc improvements to mode setting * Update changelog with PR number * Rebase against master
Added settings flag `exit_after_read` that let to terminate the pipel… …ine once it red all files. Closes logstash-plugins#212 Fixes logstash-plugins#240
Bugfixed double conversion to seconds of sincedb_clean_after Fixes logstash-plugins#257
Fix logstash-plugins#248, sinceDB now can handle pathnames with spaces This also fixes a couple of tests that previously didn't test anything due to the input IO iterator not being rewound before being passed to the code under test. Also, this updates the README to specify a missing build step. Bumped version 4.1.13 Fixes logstash-plugins#249
PreviousNext