Skip to content

Tags: jaalcorn/logstash-input-file

Tags

v4.2.2

Toggle v4.2.2's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Fix: sincedb_clean_after not being respected (logstash-plugins#276)

In certain cases, in read mode sincedb is not updated.

There was no periodic check for sincedb updates that would cause the sincedb_clean_after entries to cleanup.
The cleanup relied on new files being discovered or new content being added to existing files -> causing sincedb updates.

The fix here is to periodically flush sincedb (from the watch loop).
Besides, to make the process more deterministic, there's a minor change to make sure the same "updated" timestamp is used to mark the last changed time.

resolves logstash-plugins#250
expected to also resolve logstash-plugins#260

v4.2.1

Toggle v4.2.1's commit message
Fix: skip sincedb eviction if read mode completion deletes file durin…

…g flush

If the value of the setting `sincedb_clean_after` is too short (1 ms) it happens that during the content reading loop (in read mode) the loops itself trigger the cleanup of the same value from the SincedbCollection and then it
also try to access it again creating an error.
This commit resolves the problem guarding against the presence in the sincebd_collection before accessing it.

Fixes: logstash-plugins#272
Closes: logstash-plugins#273

v4.2.0

Toggle v4.2.0's commit message

Verified

This commit was signed with the committer’s verified signature. The key has expired.
kares Karol Bucek
Fix: watched files performance with huge filesets (logstash-plugins#268)

Plugin tracks all discovered files in a map-like collection
(`FileWatch::WatchedFilesCollection`).
The collection design was to re-sort all elements every time the
collection gets updated ... this obviously does not scale with 10000s of
watched files.
Starting up LS with 60k files in a watched directory, locally, takes
**\~ an hour before these changes**.
**After these changes** for the plugin to start processing files it
takes  **~2 seconds**.

Since the collection is known to have a noticeable (memory) footprint,
there's changes towards reducing intermediate garbage - which also
contributed to the decision to move the collection to native.

Notable changes to tracking watched files with
`FileWatch::WatchedFilesCollection`:
- predictable `log(size)` modification (and access) times
- implementation has clear locking semantics, previously operations
weren't 100% atomic
- hopefully, cleaner API as the collection re-resembles a *WatchedFile
-> String (path)* map
- the collection now needs an explicit update whenever a file changes
(on re-stat),
previous implicitness of picking up changes might have been the
reasoning behind re-sorts
(could be decoupled further by making `WatchedFile` immutable but
there's other state anyway)

---

Apart from the necessary changes to resolve the performance bottleneck,
there's also a quick review of (trace) logging - annoying to not only
see partial traces even in debug/trace levels.

Few unused methods and instance variables were removed for a better code
reading experience.

v4.1.18

Toggle v4.1.18's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Fix: release watched files on completion (in read-mode) (logstash-plu…

…gins#271)

not doing so leads to a steady increase in watched collection's size
over time (esp. in use-cases where user is pulling in new files).

the left-over file is never to be processed again - it's being deleted
anyway using the completion handler.

v4.1.17

Toggle v4.1.17's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
[DOC] Doc improvements (logstash-plugins#266)

* Doc improvements to mode setting

* Update changelog with PR number

* Rebase against master

v4.1.16

Toggle v4.1.16's commit message
Added settings flag `exit_after_read` that let to terminate the pipel…

…ine once it red all files. Closes logstash-plugins#212

Fixes logstash-plugins#240

v4.1.15

Toggle v4.1.15's commit message
Bugfixed double conversion to seconds of sincedb_clean_after

Fixes logstash-plugins#257

v4.1.14

Toggle v4.1.14's commit message
Bump version 4.1.14

Fixes logstash-plugins#255

v4.1.13

Toggle v4.1.13's commit message
Fix logstash-plugins#248, sinceDB now can handle pathnames with spaces

This also fixes a couple of tests that previously didn't test anything
due to the input IO iterator not being rewound before being passed
to the code under test.

Also, this updates the README to specify a missing build step.

Bumped version 4.1.13

Fixes logstash-plugins#249

v4.1.12

Toggle v4.1.12's commit message
CHANGELOG, gemspec: Bump to 4.1.12 (fix regression in exclude)