Skip to content

internal/testrunner/script: add script testing package #3012

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

internal/testrunner/script: add script testing package #3012

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
ab30d9e
internal/testrunner/script: add script testing package
efd6 Sep 16, 2025
9d132cf
repair
efd6 Oct 27, 2025
9825fc9
escape
efd6 Oct 27, 2025
09e3c8b
more repair
efd6 Oct 28, 2025
f72e96c
run independent stack script test in ci
efd6 Oct 28, 2025
0da1133
bump github.com/elastic/package-spec to v3.5.1
efd6 Nov 4, 2025
7588561
address pr comments: documentation
efd6 Nov 4, 2025
819f814
address pr comment: use loc.TempDir
efd6 Nov 4, 2025
a524978
address pr comment: expose PROFILE env var
efd6 Nov 5, 2025
fc8c9db
names
efd6 Nov 5, 2025
054ae48
more names
efd6 Nov 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
names
  • Loading branch information
@efd6
efd6 committed Nov 6, 2025
commit fc8c9dbb3d2dcde8eb50af95c9c9c8961b5199eb
17 changes: 9 additions & 8 deletions docs/howto/script_testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,9 @@ a stack, starting agents and services and validating results.
- `CONFIG_ROOT`: the `elastic-package` configuration root path
- `CONFIG_PROFILES`: the `elastic-package` profiles configuration root path
- `HOME`: the user's home directory path
- `PKG`: the name of the running package
- `PKG_ROOT`: the path to the root of the running package
- `PACKAGE_NAME`: the name of the running package
- `PACKAGE_BASE`: the basename of the path to the root of the running package
- `PACKAGE_ROOT`: the path to the root of the running package
- `CURRENT_VERSION`: the current version of the package
- `PREVIOUS_VERSION`: the previous version of the package
- `DATA_STREAM`: the name of the data stream
Expand All @@ -107,24 +108,24 @@ As an example, a basic system test could be expressed as follows.
[!exec:jq] skip 'Skipping test requiring absent jq command'
Copy link
Member

@jsoriano jsoriano Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like several sample tests here depend on jq to compare fields in JSON files. If this is going to be frequent, maybe we can provide a function to do these comparisons? And maybe the same with basic http requests to avoid depending on curl.

This would help to avoid depending on external tools that may not be installed, specially on Windows.

Copy link
Contributor Author

@efd6 efd6 Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it becomes necessary, those can be added. Both curl and jq are available in CI and I'd prefer to avoid having to reimplement and maintain tools that already exist.

Copy link
Member

@jsoriano jsoriano Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree this can be added later.

I am not so worried about the current CI, but about future deployments where we might forget to install jq (or curl in Windows) if not required for other things, and the test would be silently skipped. Same thing for local execution.

jq and curl are powerful tools, in no case I would think on re-implement them, but the limited logic that we need, also reusing existing tools, in the form of Go code.


# Register running stack.
use_stack -profile ${CONFIG_PROFILES}/default
use_stack -profile ${CONFIG_PROFILES}/${PROFILE}

# Install an agent.
install_agent -profile ${CONFIG_PROFILES}/default NETWORK_NAME
install_agent -profile ${CONFIG_PROFILES}/${PROFILE} NETWORK_NAME

# Bring up a docker container.
#
# The service is described in the test-hits/docker-compose.yml below with
# its logs in test-hits/logs/generated.log.
docker_up -profile ${CONFIG_PROFILES}/default -network ${NETWORK_NAME} test-hits
docker_up -profile ${CONFIG_PROFILES}/${PROFILE} -network ${NETWORK_NAME} test-hits

# Add the package resources.
add_package -profile ${CONFIG_PROFILES}/default
add_package -profile ${CONFIG_PROFILES}/${PROFILE}

# Add the data stream.
#
# The configuration for the test is described in test_config.yaml below.
add_data_stream -profile ${CONFIG_PROFILES}/default test_config.yaml DATA_STREAM_NAME
add_data_stream -profile ${CONFIG_PROFILES}/${PROFILE} test_config.yaml DATA_STREAM_NAME

# Start the service.
docker_signal test-hits SIGHUP
Expand All @@ -133,7 +134,7 @@ docker_signal test-hits SIGHUP
docker_wait_exit -timeout 5m test-hits

# Check that we can see our policy.
get_policy -profile ${CONFIG_PROFILES}/default -timeout 1m ${DATA_STREAM_NAME}
get_policy -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m ${DATA_STREAM_NAME}
cp stdout got_policy.json
exec jq '.name=="'${DATA_STREAM_NAME}'"' got_policy.json
stdout true
Expand Down
12 changes: 6 additions & 6 deletions internal/testrunner/script/agents.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,13 @@ import (
func installAgent(ts *testscript.TestScript, neg bool, args []string) {
clearStdStreams(ts)

pkgRoot := ts.Getenv("PKG_ROOT")
pkgRoot := ts.Getenv("PACKAGE_ROOT")
if pkgRoot == "" {
ts.Fatalf("PKG_ROOT is not set")
ts.Fatalf("PACKAGE_ROOT is not set")
}
pkg := ts.Getenv("PKG")
pkg := ts.Getenv("PACKAGE_NAME")
if pkg == "" {
ts.Fatalf("PKG is not set")
ts.Fatalf("PACKAGE_NAME is not set")
}
ds := ts.Getenv("DATA_STREAM")
if ds == "" {
Expand Down Expand Up @@ -158,9 +158,9 @@ func doKibanaAgent(ctx context.Context, cli *kibana.Client, fn func(a kibana.Age
func uninstallAgent(ts *testscript.TestScript, neg bool, args []string) {
clearStdStreams(ts)

pkg := ts.Getenv("PKG")
pkg := ts.Getenv("PACKAGE_NAME")
if pkg == "" {
ts.Fatalf("PKG is not set")
ts.Fatalf("PACKAGE_NAME is not set")
}
ds := ts.Getenv("DATA_STREAM")
if ds == "" {
Expand Down
12 changes: 6 additions & 6 deletions internal/testrunner/script/data_stream.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,13 @@ import (
func addDataStream(ts *testscript.TestScript, neg bool, args []string) {
clearStdStreams(ts)

pkgRoot := ts.Getenv("PKG_ROOT")
pkgRoot := ts.Getenv("PACKAGE_ROOT")
if pkgRoot == "" {
ts.Fatalf("PKG_ROOT is not set")
ts.Fatalf("PACKAGE_ROOT is not set")
}
pkg := ts.Getenv("PKG")
pkg := ts.Getenv("PACKAGE_NAME")
if pkg == "" {
ts.Fatalf("PKG is not set")
ts.Fatalf("PACKAGE_NAME is not set")
}
ds := ts.Getenv("DATA_STREAM")
if ds == "" {
Expand Down Expand Up @@ -123,9 +123,9 @@ func addDataStream(ts *testscript.TestScript, neg bool, args []string) {
func removeDataStream(ts *testscript.TestScript, neg bool, args []string) {
clearStdStreams(ts)

pkg := ts.Getenv("PKG")
pkg := ts.Getenv("PACKAGE_NAME")
if pkg == "" {
ts.Fatalf("PKG is not set")
ts.Fatalf("PACKAGE_NAME is not set")
}
ds := ts.Getenv("DATA_STREAM")
if ds == "" {
Expand Down
4 changes: 2 additions & 2 deletions internal/testrunner/script/debugging.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ var errPolicyNotFound = errors.New("not found")
func getPolicyCommand(ts *testscript.TestScript, neg bool, args []string) {
clearStdStreams(ts)

pkg := ts.Getenv("PKG")
pkg := ts.Getenv("PACKAGE_NAME")
if pkg == "" {
ts.Fatalf("PKG is not set")
ts.Fatalf("PACKAGE_NAME is not set")
}
ds := ts.Getenv("DATA_STREAM")
if ds == "" {
Expand Down
28 changes: 14 additions & 14 deletions internal/testrunner/script/package.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,15 @@ import (
func addPackage(ts *testscript.TestScript, neg bool, args []string) {
clearStdStreams(ts)

pkgRoot := ts.Getenv("PKG_ROOT")
pkgRoot := ts.Getenv("PACKAGE_ROOT")
if pkgRoot == "" {
ts.Fatalf("PKG_ROOT is not set")
ts.Fatalf("PACKAGE_ROOT is not set")
}
root, err := os.OpenRoot(pkgRoot)
ts.Check(err)
pkg := ts.Getenv("PKG")
pkg := ts.Getenv("PACKAGE_NAME")
if pkg == "" {
ts.Fatalf("PKG is not set")
ts.Fatalf("PACKAGE_NAME is not set")
}

stacks, ok := ts.Value(runningStackTag{}).(map[string]*runningStack)
Expand Down Expand Up @@ -72,15 +72,15 @@ func addPackage(ts *testscript.TestScript, neg bool, args []string) {
func removePackage(ts *testscript.TestScript, neg bool, args []string) {
clearStdStreams(ts)

pkgRoot := ts.Getenv("PKG_ROOT")
pkgRoot := ts.Getenv("PACKAGE_ROOT")
if pkgRoot == "" {
ts.Fatalf("PKG_ROOT is not set")
ts.Fatalf("PACKAGE_ROOT is not set")
}
root, err := os.OpenRoot(pkgRoot)
ts.Check(err)
pkg := ts.Getenv("PKG")
pkg := ts.Getenv("PACKAGE_NAME")
if pkg == "" {
ts.Fatalf("PKG is not set")
ts.Fatalf("PACKAGE_NAME is not set")
}

stacks, ok := ts.Value(runningStackTag{}).(map[string]*runningStack)
Expand Down Expand Up @@ -124,9 +124,9 @@ func removePackage(ts *testscript.TestScript, neg bool, args []string) {
func upgradePackageLatest(ts *testscript.TestScript, neg bool, args []string) {
clearStdStreams(ts)

pkg := ts.Getenv("PKG")
pkg := ts.Getenv("PACKAGE_NAME")
if pkg == "" {
ts.Fatalf("PKG is not set")
ts.Fatalf("PACKAGE_NAME is not set")
}

stacks, ok := ts.Value(runningStackTag{}).(map[string]*runningStack)
Expand Down Expand Up @@ -184,9 +184,9 @@ func upgradePackageLatest(ts *testscript.TestScript, neg bool, args []string) {
func addPackageZip(ts *testscript.TestScript, neg bool, args []string) {
clearStdStreams(ts)

pkg := ts.Getenv("PKG")
pkg := ts.Getenv("PACKAGE_NAME")
if pkg == "" {
ts.Fatalf("PKG is not set")
ts.Fatalf("PACKAGE_NAME is not set")
}

stacks, ok := ts.Value(runningStackTag{}).(map[string]*runningStack)
Expand Down Expand Up @@ -228,9 +228,9 @@ func addPackageZip(ts *testscript.TestScript, neg bool, args []string) {
func removePackageZip(ts *testscript.TestScript, neg bool, args []string) {
clearStdStreams(ts)

pkg := ts.Getenv("PKG")
pkg := ts.Getenv("PACKAGE_NAME")
if pkg == "" {
ts.Fatalf("PKG is not set")
ts.Fatalf("PACKAGE_NAME is not set")
}

stacks, ok := ts.Value(runningStackTag{}).(map[string]*runningStack)
Expand Down
9 changes: 7 additions & 2 deletions internal/testrunner/script/script.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,8 +218,13 @@ func Run(dst io.Writer, cmd *cobra.Command, args []string) error {
e.Setenv("CONFIG_PROFILES", loc.ProfileDir())
e.Setenv("HOME", home)
if pkgRoot != "" {
e.Setenv("PKG", filepath.Base(pkgRoot))
e.Setenv("PKG_ROOT", pkgRoot)
m, err := packages.ReadPackageManifestFromPackageRoot(pkgRoot)
if err != nil {
return err
}
e.Setenv("PACKAGE_NAME", m.Name)
e.Setenv("PACKAGE_BASE", filepath.Base(pkgRoot))
e.Setenv("PACKAGE_ROOT", pkgRoot)
}
if currVersion != "" {
e.Setenv("CURRENT_VERSION", currVersion)
Expand Down
2 changes: 1 addition & 1 deletion test/packages/other/with_script/data_stream/first/_dev/test/scripts/agent_up_down.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ cmp stdout want_down.text
# Uninstall the agent.
uninstall_agent
! stderr .
stdout '^deleted agent policies for '${PKG}/${DATA_STREAM}
stdout '^deleted agent policies for '${PACKAGE_NAME}/${DATA_STREAM}

# Take down stack.
stack_down
Expand Down
2 changes: 1 addition & 1 deletion ...packages/other/with_script/data_stream/first/_dev/test/scripts/agent_up_down_external.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ cmp stdout want_down.text
# Uninstall the agent.
uninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE}
! stderr .
stdout '^deleted agent policies for '${PKG}/${DATA_STREAM}
stdout '^deleted agent policies for '${PACKAGE_NAME}/${DATA_STREAM}

# Then attempt to take it down again.
! stack_down -profile ${CONFIG_PROFILES}/${PROFILE}
Expand Down
4 changes: 2 additions & 2 deletions test/packages/other/with_script/data_stream/first/_dev/test/scripts/env.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ stdout '/\.elastic-package$'
exec echo ${CONFIG_PROFILES}
stdout '/\.elastic-package/profiles$'

exec echo ${PKG}
exec echo ${PACKAGE_NAME}
stdout '^with_script$'

exec echo ${PKG_ROOT}
exec echo ${PACKAGE_ROOT}
stdout '/elastic/elastic-package/test/packages/other/with_script$'

exec echo ${DATA_STREAM}
Expand Down
8 changes: 4 additions & 4 deletions test/packages/other/with_script/data_stream/first/_dev/test/scripts/get_docs_external.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ cp stdout got_docs.json
# Let's upgrade!
upgrade_package_latest -profile ${CONFIG_PROFILES}/${PROFILE}
! stderr .
stdout 'upgraded package '${PKG}' from version '${CURRENT_VERSION@R}
stdout 'upgraded package '${PACKAGE_NAME}' from version '${CURRENT_VERSION@R}

# Remove the data stream.
remove_data_stream -profile ${CONFIG_PROFILES}/${PROFILE} ${DATA_STREAM_NAME}
Expand All @@ -72,7 +72,7 @@ cmpenv stdout want_remove_data_stream.text

# Uninstall the agent.
uninstall_agent -profile ${CONFIG_PROFILES}/${PROFILE} -timeout 1m
stdout '^deleted agent policies for '${PKG}/${DATA_STREAM}
stdout '^deleted agent policies for '${PACKAGE_NAME}/${DATA_STREAM}
! stderr .

# Remove the package resources.
Expand Down Expand Up @@ -128,6 +128,6 @@ data_stream:
-- want_data_stream_name.pattern --
logs-with_script\.first-[0-9]+
-- want_add_data_stream.text --
added ${DATA_STREAM_NAME} data stream policy templates for ${PKG}/${DATA_STREAM}
added ${DATA_STREAM_NAME} data stream policy templates for ${PACKAGE_NAME}/${DATA_STREAM}
-- want_remove_data_stream.text --
removed ${DATA_STREAM_NAME} data stream policy templates for ${PKG}/${DATA_STREAM}
removed ${DATA_STREAM_NAME} data stream policy templates for ${PACKAGE_NAME}/${DATA_STREAM}