Skip to content

Commit ff56b13

Browse files
committed
Create tool to create/update geoipdatabased based on MaxMind-DB repo
1 parent 8ae8411 commit ff56b13

File tree

6 files changed

+266
-166
lines changed

6 files changed

+266
-166
lines changed

go.mod

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,14 @@ require (
2828
github.com/hashicorp/go-retryablehttp v0.7.7
2929
github.com/jedib0t/go-pretty v4.3.0+incompatible
3030
github.com/magefile/mage v1.15.0
31+
github.com/maxmind/mmdbwriter v1.0.0
3132
github.com/mholt/archiver/v3 v3.5.1
3233
github.com/olekukonko/tablewriter v0.0.5
3334
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
3435
github.com/shirou/gopsutil/v3 v3.24.5
3536
github.com/spf13/cobra v1.9.1
3637
github.com/stretchr/testify v1.10.0
38+
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba
3739
golang.org/x/tools v0.30.0
3840
gopkg.in/dnaeon/go-vcr.v3 v3.2.0
3941
gopkg.in/yaml.v3 v3.0.1
@@ -124,6 +126,7 @@ require (
124126
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
125127
github.com/nwaples/rardecode v1.1.3 // indirect
126128
github.com/oklog/ulid v1.3.1 // indirect
129+
github.com/oschwald/maxminddb-golang v1.12.0 // indirect
127130
github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
128131
github.com/pierrec/lz4/v4 v4.1.17 // indirect
129132
github.com/pkg/errors v0.9.1 // indirect

go.sum

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,8 @@ github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D
234234
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
235235
github.com/mattn/go-runewidth v0.0.14 h1:+xnbZSEeDbOIg5/mE6JF0w6n9duR1l3/WmbinWVwUuU=
236236
github.com/mattn/go-runewidth v0.0.14/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
237+
github.com/maxmind/mmdbwriter v1.0.0 h1:bieL4P6yaYaHvbtLSwnKtEvScUKKD6jcKaLiTM3WSMw=
238+
github.com/maxmind/mmdbwriter v1.0.0/go.mod h1:noBMCUtyN5PUQ4H8ikkOvGSHhzhLok51fON2hcrpKj8=
237239
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
238240
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d h1:5PJl274Y63IEHC+7izoQE9x6ikvDFZS2mDVS3drnohI=
239241
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
@@ -276,6 +278,8 @@ github.com/onsi/ginkgo/v2 v2.21.0 h1:7rg/4f3rB88pb5obDgNZrNHrQ4e6WpjonchcpuBRnZM
276278
github.com/onsi/ginkgo/v2 v2.21.0/go.mod h1:7Du3c42kxCUegi0IImZ1wUQzMBVecgIHjR1C+NkhLQo=
277279
github.com/onsi/gomega v1.35.1 h1:Cwbd75ZBPxFSuZ6T+rN/WCb/gOc6YgFBXLlZLhC7Ds4=
278280
github.com/onsi/gomega v1.35.1/go.mod h1:PvZbdDc8J6XJEpDK4HCuRBm8a6Fzp9/DmhC9C7yFlog=
281+
github.com/oschwald/maxminddb-golang v1.12.0 h1:9FnTOD0YOhP7DGxGsq4glzpGy5+w7pq50AS6wALUMYs=
282+
github.com/oschwald/maxminddb-golang v1.12.0/go.mod h1:q0Nob5lTCqyQ8WT6FYgS1L7PXKVVbgiymefNwIjPzgY=
279283
github.com/otiai10/copy v1.14.1 h1:5/7E6qsUMBaH5AnQ0sSLzzTg1oTECmcCmT6lvF45Na8=
280284
github.com/otiai10/copy v1.14.1/go.mod h1:oQwrEDDOci3IM8dJF0d8+jnbfPDllW6vUjNc3DoZm9I=
281285
github.com/otiai10/mint v1.6.3 h1:87qsV/aw1F5as1eH1zS/yqHY85ANKVMgkDrf9rcxbQs=
@@ -369,6 +373,8 @@ go.mongodb.org/mongo-driver v1.11.1 h1:QP0znIRTuL0jf1oBQoAoM0C6ZJfBK4kx0Uumtv1A7
369373
go.mongodb.org/mongo-driver v1.11.1/go.mod h1:s7p5vEtfbeR1gYi6pnj3c3/urpbLv2T5Sfd6Rp2HBB8=
370374
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
371375
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
376+
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba h1:0b9z3AuHCjxk0x/opv64kcgZLBseWJUpBw5I82+2U4M=
377+
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
372378
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
373379
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
374380
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=

internal/stack/_static/geoip_source/README.md

Lines changed: 11 additions & 166 deletions
Original file line numberDiff line numberDiff line change
@@ -21,139 +21,10 @@ In the following sections, it is described how to build your own custom GeoIP da
2121

2222
## Prerequisites
2323

24-
1. Clone https://github.com/maxmind/MaxMind-DB
25-
2. Install GoLang 1.23 or later
26-
3. Install `mmdbinspect` and add it your PATH (https://github.com/maxmind/mmdbinspect)
27-
28-
The following section describes the changes required to build `write-test-data` tool for the `elastic-package` needs.
29-
30-
### Required changes to be done in MaxMind-DB repository
31-
32-
Before building `write-test-data` tool, it requires to apply some changes in the code
33-
to allow creating our own `elastic-package` databases.
34-
35-
These changes have been tested using the code from [this commit](https://github.com/maxmind/MaxMind-DB/commit/0ec71808b19669e9e1bf5e63a8c83b202d9bd115).
36-
37-
Changes to be applied:
38-
- `pkg/writer/geoip2.go`:
39-
- Include usage of reserved networks to allow adding documentation ranges:
40-
```diff
41-
--- pkg/writer/geoip2.go
42-
+++ pkg/writer/geoip2.go
43-
@@ -47,12 +35,13 @@ func (w *Writer) WriteGeoIP2TestDB() error {
44-
45-
dbWriter, err := mmdbwriter.New(
46-
mmdbwriter.Options{
47-
- DatabaseType: dbType,
48-
- Description: description,
49-
- DisableIPv4Aliasing: false,
50-
- IPVersion: 6,
51-
- Languages: languages,
52-
- RecordSize: 28,
53-
+ DatabaseType: dbType,
54-
+ Description: description,
55-
+ DisableIPv4Aliasing: false,
56-
+ IPVersion: 6,
57-
+ Languages: languages,
58-
+ RecordSize: 28,
59-
+ IncludeReservedNetworks: true,
60-
},
61-
)
62-
if err != nil {
63-
```
64-
- Just create GeoLite2 databases:
65-
```diff
66-
--- pkg/writer/geoip2.go
67-
+++ pkg/writer/geoip2.go
68-
@@ -16,18 +16,6 @@ import (
69-
// WriteGeoIP2TestDB writes GeoIP2 test mmdb files.
70-
func (w *Writer) WriteGeoIP2TestDB() error {
71-
dbTypes := []string{
72-
- "GeoIP2-Anonymous-IP",
73-
- "GeoIP2-City",
74-
- "GeoIP2-Connection-Type",
75-
- "GeoIP2-Country",
76-
- "GeoIP2-DensityIncome",
77-
- "GeoIP2-Domain",
78-
- "GeoIP2-Enterprise",
79-
- "GeoIP2-IP-Risk",
80-
- "GeoIP2-ISP",
81-
- "GeoIP2-Precision-Enterprise",
82-
- "GeoIP2-Static-IP-Score",
83-
- "GeoIP2-User-Count",
84-
"GeoLite2-ASN",
85-
"GeoLite2-City",
86-
"GeoLite2-Country",
87-
```
88-
- `cmd/write-test-data/main.go`
89-
- Just trigger the creation of GeoLite2 databases:
90-
```diff
91-
--- cmd/write-test-data/main.go
92-
+++ cmd/write-test-data/main.go
93-
@@ -21,46 +21,6 @@ func main() {
94-
os.Exit(1)
95-
}
96-
97-
- if err := w.WriteIPv4TestDB(); err != nil {
98-
- fmt.Printf("writing IPv4 test databases: %+v\n", err)
99-
- os.Exit(1)
100-
- }
101-
-
102-
- if err := w.WriteIPv6TestDB(); err != nil {
103-
- fmt.Printf("writing IPv6 test databases: %+v\n", err)
104-
- os.Exit(1)
105-
- }
106-
-
107-
- if err := w.WriteMixedIPTestDB(); err != nil {
108-
- fmt.Printf("writing IPv6 test databases: %+v\n", err)
109-
- os.Exit(1)
110-
- }
111-
-
112-
- if err := w.WriteNoIPv4TestDB(); err != nil {
113-
- fmt.Printf("writing no IPv4 test databases: %+v\n", err)
114-
- os.Exit(1)
115-
- }
116-
-
117-
- if err := w.WriteNoMapTestDB(); err != nil {
118-
- fmt.Printf("writing no map test databases: %+v\n", err)
119-
- os.Exit(1)
120-
- }
121-
-
122-
- if err := w.WriteMetadataPointersTestDB(); err != nil {
123-
- fmt.Printf("writing metadata pointers test databases: %+v\n", err)
124-
- os.Exit(1)
125-
- }
126-
-
127-
- if err := w.WriteDecoderTestDB(); err != nil {
128-
- fmt.Printf("writing decoder test databases: %+v\n", err)
129-
- os.Exit(1)
130-
- }
131-
-
132-
- if err := w.WriteDeeplyNestedStructuresTestDB(); err != nil {
133-
- fmt.Printf("writing decoder test databases: %+v\n", err)
134-
- os.Exit(1)
135-
- }
136-
-
137-
if err := w.WriteGeoIP2TestDB(); err != nil {
138-
fmt.Printf("writing GeoIP2 test databases: %+v\n", err)
139-
os.Exit(1)
140-
```
141-
142-
Once applied all these changes, you can build the `write-test-data` tool:
143-
```shell
144-
git clone https://github.com/maxmind/MaxMind-DB
145-
146-
cd MaxMind-DB
147-
cd cmd/write-test-data
24+
1. Install GoLang 1.23 or later
25+
2. Install `mmdbinspect` and add it your PATH (https://github.com/maxmind/mmdbinspect)
14826

149-
# Before building it, apply the changes mentioned in this section.
150-
151-
go build
152-
153-
# A new binary `write-test-data` should have been generated in the same directory.
154-
```
155-
156-
This tool is the one to be used in the following section.
27+
The following section describes the procedure to generate new `mmdb` databases from the JSON source files.
15728

15829
## How to build a new GeoIP database for `elastic-package`
15930

@@ -165,51 +36,25 @@ The latest JSON Files used to generate the `mmdb` databases are located in `inte
16536
Those JSON files already contain the required entries for the documentation ranges with some dummy GeoIP data. And they should be
16637
used as a basis for new changes if required.
16738

168-
As an example, the following example shows how to generate new GeoIP databases (`*.mmdb`) using the source JSON
169-
files (`source-data` folder) defined in the [MaxMind-DB](https://github.com/maxmind/MaxMind-DB) repository without any modification:
170-
171-
```shell
172-
cd path/to/repo/MaxMind-DB
173-
# NOTE: Let's consider that the binary has already been built, and it is available
174-
# at `cmd/write-test-data/write-test-data`
175-
176-
# This creates the target folder, and it will replace any file that
177-
# could exist in the folder
178-
cmd/write-test-data/write-test-data -source source-data -target my-target-data
179-
```
180-
181-
As mentioned above, `elastic-package` requires to add new entries to add data for the
182-
documentation ranges.
183-
184-
If any other changes are required in the GeoIP databases used by elastic-package, those JSON files located at `internal/stack/_static/geoip_sources`
185-
can be updated and then new `mmdb` files be generated:
39+
If any other changes are required in the GeoIP databases used by elastic-package, update the JSON files located at `internal/stack/_static/geoip_sources`
40+
and then generate new `mmdb` files:
18641
```shell
18742
cd path/to/repo/elastic-package
188-
cd internal/stack/_static
189-
190-
# 1. Add the required data into the JSON files in `geoip_sources`
191-
# 2. Generate mmdb databases with the tool built previously
192-
path/to/cmd/write-test-data/write-test-data -source geoip_sources -target new_databases
19343

194-
# 3. Copy the new databases to the directory that `elastic-package` expects
195-
mv new_databases/GeoLite2-ASN-Test.mmdb GeoLite2-ASN.mmdb
196-
mv new_databases/GeoLite2-Country-Test.mmdb GeoLite2-Country.mmdb
197-
mv new_databases/GeoLite2-City-Test.mmdb GeoLite2-City.mmdb
44+
# 1. Add the required data into the JSON files in `geoip_sources` (internal/stack/_static/geoip_sources)
45+
# 2. Generate mmdb databases
46+
go run ./tools/geoipdatabases -source internal/stack/_static/geoip_sources -target internal/stack/_static
19847

199-
# 4. Remove new_databases folder
200-
rm -rf new_databases
20148
```
20249

20350
Those databases generated can be tested to ensure that the expected data is returned using the `mmdbinspect` tool.
20451
For instance:
20552
```shell
206-
mmdbinspect -db internal/stack/_static/GeoLite2-ASN-Test.mmdb 192.0.2.100
207-
mmdbinspect -db internal/stack/_static/GeoLite2-Country-Test.mmdb 192.0.2.100
208-
mmdbinspect -db internal/stack/_static/GeoLite2-City-Test.mmdb 192.0.2.100
53+
mmdbinspect -db internal/stack/_static/GeoLite2-ASN.mmdb 192.0.2.100
54+
mmdbinspect -db internal/stack/_static/GeoLite2-Country.mmdb 192.0.2.100
55+
mmdbinspect -db internal/stack/_static/GeoLite2-City.mmdb 192.0.2.100
20956
```
21057

211-
Take into account that the files generated by `write-test-data` must be renamed to match the file names used by `elastic-package`.
212-
21358
As these GeoIP databases are installed when the stack is bootstrapped, any change in those databases require to
21459
restart the stack:
21560
```shell

0 commit comments

Comments
 (0)