Skip to content

Cog 793 metadata rework #460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Vasilije1990 merged 10 commits into from
Jan 23, 2025
Merged

Cog 793 metadata rework #460

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
0c7c1d7
refactor: Refactor ingestion to only have one ingestion task
dexters1 Jan 20, 2025
49ad292
refactor: Reduce complexity of metadata handling
dexters1 Jan 20, 2025
ab8d95c
refactor: As neo4j can't support dictionaries, add foreign metadata a…
dexters1 Jan 20, 2025
5c17501
refactor: add missing foreing_metadata attr to tests
dexters1 Jan 20, 2025
4196a4c
refactor: Update test to be up to date with current metadata refactor…
dexters1 Jan 20, 2025
77f0b45
refactor: Resolve issue with notebook after metadata refactor
dexters1 Jan 20, 2025
bd3a5a7
Merge branch 'dev' into COG-793-metadata-rework
dexters1 Jan 20, 2025
655ab0b
Merge branch 'dev' into COG-793-metadata-rework
dexters1 Jan 21, 2025
80e67b0
refactor: Rename foreign to external metadata
dexters1 Jan 22, 2025
40c0279
Merge branch 'COG-793-metadata-rework' of github.com:topoteretes/cogn…
dexters1 Jan 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor: add missing foreing_metadata attr to tests
  • Loading branch information
@dexters1
dexters1 committed Jan 20, 2025
commit 5c17501bb8d3825214e23fa3ea9cfbdd835d12f1
1 change: 1 addition & 0 deletions cognee/tests/integration/documents/AudioDocument_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ def test_AudioDocument():
id=uuid.uuid4(),
name="audio-dummy-test",
raw_data_location="",
foreign_metadata="",
mime_type="",
)
with patch.object(AudioDocument, "create_transcript", return_value=TEST_TEXT):
Expand Down
1 change: 1 addition & 0 deletions cognee/tests/integration/documents/ImageDocument_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ def test_ImageDocument():
id=uuid.uuid4(),
name="image-dummy-test",
raw_data_location="",
foreign_metadata="",
mime_type="",
)
with patch.object(ImageDocument, "transcribe_image", return_value=TEST_TEXT):
Expand Down
1 change: 1 addition & 0 deletions cognee/tests/integration/documents/TextDocument_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ def test_TextDocument(input_file, chunk_size):
id=uuid.uuid4(),
name=input_file,
raw_data_location=test_file_path,
foreign_metadata="",
Copy link
Contributor

@coderabbitai coderabbitai bot Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add test cases for foreign_metadata handling.

The test currently initializes foreign_metadata with an empty string but doesn't verify how the document handles various foreign metadata scenarios. Consider adding test cases for:

  • Valid JSON metadata
  • Empty metadata
  • Invalid JSON metadata

Example test case:

@pytest.mark.parametrize(
    "foreign_metadata,expected_result",
    [
        ('{"source": "external", "tags": ["test"]}', True),
        ("", True),
        ("invalid json", False),
    ],
)
def test_TextDocument_foreign_metadata(foreign_metadata, expected_result):
    document = TextDocument(
        id=uuid.uuid4(),
        name="test.txt",
        raw_data_location="test_path",
        foreign_metadata=foreign_metadata,
        mime_type=""
    )
    assert hasattr(document, "foreign_metadata") == expected_result

mime_type="",
)

Expand Down
4 changes: 4 additions & 0 deletions cognee/tests/integration/documents/UnstructuredDocument_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,27 +39,31 @@ def test_UnstructuredDocument():
id=uuid.uuid4(),
name="example.pptx",
raw_data_location=pptx_file_path,
foreign_metadata="",
mime_type="application/vnd.openxmlformats-officedocument.presentationml.presentation",
)

docx_document = UnstructuredDocument(
id=uuid.uuid4(),
name="example.docx",
raw_data_location=docx_file_path,
foreign_metadata="",
mime_type="application/vnd.openxmlformats-officedocument.wordprocessingml.document",
)

csv_document = UnstructuredDocument(
id=uuid.uuid4(),
name="example.csv",
raw_data_location=csv_file_path,
foreign_metadata="",
mime_type="text/csv",
)

xlsx_document = UnstructuredDocument(
id=uuid.uuid4(),
name="example.xlsx",
raw_data_location=xlsx_file_path,
foreign_metadata="",
mime_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
)

Expand Down
Loading