[8.19](backport #50137) filestream: Fix shutdown logic and improve benchmark

[mergify]

Proposed commit message

Fix filestream benchmark correctness and shutdown, add high-file-count sub-benchmarks

Shutdown fix (notifyObserver):

During shutdown the watcher goroutine that drains notifyChan exits before
harvesters finish. The old blocking send in notifyObserver stalled every
closing harvester until the task group's 1-minute timeout expired. Replace the
blocking send with a select on canceler.Done() so harvesters unblock
immediately when the input is cancelled.

Benchmark fixes (correctness):

The inode-mode benchmarks were silently broken: file_identity defaulted to
fingerprint even though prospector.scanner.fingerprint.enabled was false,
so every file received the same empty-fingerprint identity and only one
harvester was started out of N files. Explicitly set file_identity.native /
file_identity.fingerprint to match the scanner mode so each file gets its own
identity and harvester.

Benchmark fixes (hangs / timeouts):

Without close.reader.on_eof: true harvesters waited for more data after EOF,
preventing the pipeline from closing until the 60-second task.Group.Stop
timeout expired. Combined with a 1-second check_interval this made multi-file
benchmarks extremely slow. Set close.reader.on_eof: true and lower
check_interval to 100 ms so harvesters close promptly.

Benchmark refactoring:

• Consolidate duplicated single-file / multi-file / inode / fingerprint
  sub-benchmarks into a table-driven loop.
• Add 1 000-file and 10 000-file fingerprint sub-benchmarks to stress per-file
  overhead (logger cloning, reader pipeline setup, fingerprint I/O).
• Replace deprecated logging setup with local loggers.
• Only buffer the event channel when events are actually collected, avoiding a
  10 000-slot channel allocation in benchmarks that discard events.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

How to test this PR locally

cd filebeat
go test -bench BenchmarkFilestream -benchtime 1x ./input/filestream/...
```<hr>This is an automatic backport of pull request #50137 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 2977528 has failed:

On branch mergify/bp/8.19/pr-50137
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 297752842.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   filebeat/input/filestream/internal/input-logfile/harvester.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   filebeat/input/filestream/input_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[github-actions]

TL;DR

The failing Buildkite step is caused by unresolved Git conflict markers committed in filebeat/input/filestream/input_test.go, so pre-commit fails on check-merge-conflict. Resolve that file’s cherry-pick conflict and push the corrected backport commit.

Remediation

• Resolve filebeat/input/filestream/input_test.go by removing conflict markers and keeping the intended merged benchmark/helper code (filestreamBenchCfg + surrounding BenchmarkFilestream block).
• Re-run pre-commit run --all-files (or restart Buildkite) to confirm check-merge-conflict passes before re-requesting review.

Investigation details

Root Cause

pre-commit failed in agentbeat: Run pre-commit because check-merge-conflict detected conflict markers in filebeat/input/filestream/input_test.go.

The PR head content at commit 1d2117afad16753a359e3ef843c6f4767ae5f30e still contains:

• <<<<<<< HEAD
• =======
• >>>>>>> 297752842 (filestream: Fix shutdown logic and improve benchmark (#50137))

This is consistent with the backport conflict note already present on the PR (both modified: filebeat/input/filestream/input_test.go).

Evidence

• Build: https://buildkite.com/elastic/beats/builds/44297
• Job/step: agentbeat: Run pre-commit → pre-commit run --all-files
• Key log excerpt:

check for merge conflicts................................................Failed
- hook id: check-merge-conflict
filebeat/input/filestream/input_test.go:118: Merge conflict string '<<<<<<<' found
filebeat/input/filestream/input_test.go:119: Merge conflict string '=======' found
filebeat/input/filestream/input_test.go:138: Merge conflict string '>>>>>>>' found

Verification

• Not run locally (analysis-only workflow). Failure is directly evidenced by Buildkite log + PR head file content.

Follow-up

After resolving and pushing, this Buildkite failure should clear; any remaining failures (if present) can then be analyzed independently.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @orestisfl? 🙏