This section describes how to build and query a graph of the social network data in Neo4j. It uses the official neo4j Python client to perform the ingestion and querying.
Because Neo4j uses a client-server architecture, for development purposes, it makes sense to use Docker to orchestrate the setup and teardown of the DB. This is done easily via docker-compose as follows.
The necessary authentication to the database is specified via the variables in .env.example. Copy this example file, rename it to .env and update the NEO4J_PASSWORD field with the desired DB password.
Then, run the Docker container in detached mode as follows.
docker compose up -dOnce development and querying are finished, the container can be stopped as follows.
docker compose downNote
All timing numbers shown below are on an M3 Macbook Pro with 32 GB of RAM.
The script build_graph.py contains the necessary methods to connect to the Neo4j DB and ingest the data from the CSV files, in batches for large amounts of data.
python build_graph.pyThe numbers shown below are for when we ingest 100K person nodes, ~10K location nodes and ~2. 4M edges into the graph. Note the following points:
- The goal is to perform the entire task in Python, so we don't want to use other means like
apocotLOAD CSVto ingest the data (which may be faster, but would require additional glue code, which defeats the purpose of this exercise) - The async API of the Neo4j Python client is used, which is observed on this dataset to perform ~40% faster than the sync API
- The person nodes and person-person follower edges are ingested in batches, which is part of the best practices when passing data to Neo4j via Python -- this is because the number of persons and followers can get very large, causing the number of edges to nonlinearly increase with the size of the dataset.
- The batch size is set to 500K, which may seem large at first glance, but for the given data, the nodes and edges, even after
UNWINDing in Cypher, are small enough to fit in batch memory per transaction -- the memory requirements may be different on more complex datasets
# Set large batch size of 500k
$ python build_graph.py -b 500000
Nodes loaded in 2.3581s
Edges loaded in 30.8509sAs expected, the nodes load much faster than the edges, since there are many more edges than nodes. In addition, the nodes in Neo4j are indexed (via uniqueness constraints), following which the edges are created based on a match on existing nodes, allowing us to achieve this performance.
The script query.py contains a suite of queries that can be run to benchmark various aspects of the DB's performance.
python query.pyQuery 1:
MATCH (follower:Person)-[:FOLLOWS]->(person:Person)
RETURN person.personID AS personID, person.name AS name, count(follower) AS numFollowers
ORDER BY numFollowers DESC LIMIT 3
Top 3 most-followed persons:
shape: (3, 3)
┌──────────┬───────────────────┬──────────────┐
│ personID ┆ name ┆ numFollowers │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ i64 │
╞══════════╪═══════════════════╪══════════════╡
│ 85723 ┆ Melissa Murphy ┆ 4998 │
│ 68753 ┆ Jocelyn Patterson ┆ 4985 │
│ 54696 ┆ Michael Herring ┆ 4976 │
└──────────┴───────────────────┴──────────────┘
Query 2:
MATCH (follower:Person) -[:FOLLOWS]-> (person:Person)
WITH person, count(follower) as followers
ORDER BY followers DESC LIMIT 1
MATCH (person) -[:LIVES_IN]-> (city:City)
RETURN person.name AS name, followers AS numFollowers, city.city AS city, city.state AS state, city.country AS country
City in which most-followed person lives:
shape: (1, 5)
┌────────────────┬──────────────┬────────┬───────┬───────────────┐
│ name ┆ numFollowers ┆ city ┆ state ┆ country │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ str ┆ str ┆ str │
╞════════════════╪══════════════╪════════╪═══════╪═══════════════╡
│ Melissa Murphy ┆ 4998 ┆ Austin ┆ Texas ┆ United States │
└────────────────┴──────────────┴────────┴───────┴───────────────┘
Query 3:
MATCH (p:Person) -[:LIVES_IN]-> (c:City) -[*1..2]-> (co:Country)
WHERE co.country = $country
RETURN c.city AS city, avg(p.age) AS averageAge
ORDER BY averageAge LIMIT 5
Cities with lowest average age in United States:
shape: (5, 2)
┌─────────────┬────────────┐
│ city ┆ averageAge │
│ --- ┆ --- │
│ str ┆ f64 │
╞═════════════╪════════════╡
│ Austin ┆ 37.655491 │
│ Kansas City ┆ 37.742365 │
│ Miami ┆ 37.7763 │
│ San Antonio ┆ 37.810841 │
│ Houston ┆ 37.817708 │
└─────────────┴────────────┘
Query 4:
MATCH (p:Person)-[:LIVES_IN]->(ci:City)-[*1..2]->(country:Country)
WHERE p.age >= $age_lower AND p.age <= $age_upper
RETURN country.country AS countries, count(country) AS personCounts
ORDER BY personCounts DESC LIMIT 3
Persons between ages 30-40 in each country:
shape: (3, 2)
┌────────────────┬──────────────┐
│ countries ┆ personCounts │
│ --- ┆ --- │
│ str ┆ i64 │
╞════════════════╪══════════════╡
│ United States ┆ 30733 │
│ Canada ┆ 3046 │
│ United Kingdom ┆ 1816 │
└────────────────┴──────────────┘
Query 5:
MATCH (p:Person)-[:HAS_INTEREST]->(i:Interest)
WHERE tolower(i.interest) = tolower($interest)
AND tolower(p.gender) = tolower($gender)
WITH p, i
MATCH (p)-[:LIVES_IN]->(c:City)
WHERE c.city = $city AND c.country = $country
RETURN count(p) AS numPersons
Number of male users in London, United Kingdom who have an interest in fine dining:
shape: (1, 1)
┌────────────┐
│ numPersons │
│ --- │
│ i64 │
╞════════════╡
│ 52 │
└────────────┘
Query 6:
MATCH (p:Person)-[:HAS_INTEREST]->(i:Interest)
WHERE tolower(i.interest) = tolower($interest)
AND tolower(p.gender) = tolower($gender)
WITH p, i
MATCH (p)-[:LIVES_IN]->(c:City)
RETURN count(p) AS numPersons, c.city AS city, c.country AS country
ORDER BY numPersons DESC LIMIT 5
Cities with the most female users who have an interest in tennis:
shape: (5, 3)
┌────────────┬────────────┬────────────────┐
│ numPersons ┆ city ┆ country │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str │
╞════════════╪════════════╪════════════════╡
│ 66 ┆ Houston ┆ United States │
│ 66 ┆ Birmingham ┆ United Kingdom │
│ 65 ┆ Raleigh ┆ United States │
│ 64 ┆ Montreal ┆ Canada │
│ 62 ┆ Phoenix ┆ United States │
└────────────┴────────────┴────────────────┘
Query 7:
MATCH (p:Person)-[:LIVES_IN]->(:City)-[:CITY_IN]->(s:State)
WHERE p.age >= $age_lower AND p.age <= $age_upper AND s.country = $country
WITH p, s
MATCH (p)-[:HAS_INTEREST]->(i:Interest)
WHERE tolower(i.interest) = tolower($interest)
RETURN count(p) AS numPersons, s.state AS state, s.country AS country
ORDER BY numPersons DESC LIMIT 1
State in United States with the most users between ages 23-30 who have an interest in photography:
shape: (1, 3)
┌────────────┬────────────┬───────────────┐
│ numPersons ┆ state ┆ country │
│ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str │
╞════════════╪════════════╪═══════════════╡
│ 168 ┆ California ┆ United States │
└────────────┴────────────┴───────────────┘
Query 8:
MATCH (a:Person)-[r1:FOLLOWS]->(b:Person)-[r2:FOLLOWS]->(c:Person)
RETURN count(*) AS numPaths
Number of second-degree paths:
shape: (1, 1)
┌──────────┐
│ numPaths │
│ --- │
│ i64 │
╞══════════╡
│ 58431994 │
└──────────┘
Query 9:
MATCH (a:Person)-[r1:FOLLOWS]->(b:Person)-[r2:FOLLOWS]->(c:Person)
WHERE b.age < $age_1 AND c.age > $age_2
RETURN count(*) as numPaths
Number of paths through persons below 50 to persons above 25:
shape: (1, 1)
┌──────────┐
│ numPaths │
│ --- │
│ i64 │
╞══════════╡
│ 46220422 │
└──────────┘
Neo4j query script completed in 9.079268s
The benchmark is run using pytest-benchmark package as follows.
$ pytest benchmark_query.py --benchmark-min-rounds=5 --benchmark-warmup-iterations=5 --benchmark-disable-gc --benchmark-sort=fullname
================================================= test session starts ==================================================
platform darwin -- Python 3.11.7, pytest-8.1.1, pluggy-1.4.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=True min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=5)
rootdir: /Users/prrao/code/kuzudb-study/neo4j
plugins: Faker-23.1.0, benchmark-4.0.0
collected 9 items
benchmark_query.py ......... [100%]
--------------------------------------------------------------------------------- benchmark: 9 tests ---------------------------------------------------------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_query1 1.6634 (249.15) 1.8413 (171.27) 1.7614 (219.26) 0.0710 (78.24) 1.7787 (229.50) 0.1107 (185.42) 2;0 0.5677 (0.00) 5 1
test_benchmark_query2 0.5965 (89.35) 0.6333 (58.91) 0.6149 (76.55) 0.0160 (17.68) 0.6091 (78.59) 0.0276 (46.27) 2;0 1.6262 (0.01) 5 1
test_benchmark_query3 0.0360 (5.39) 0.0463 (4.30) 0.0388 (4.83) 0.0023 (2.49) 0.0384 (4.96) 0.0020 (3.37) 4;1 25.7565 (0.21) 20 1
test_benchmark_query4 0.0404 (6.04) 0.0500 (4.65) 0.0426 (5.30) 0.0023 (2.55) 0.0421 (5.43) 0.0017 (2.90) 3;3 23.4888 (0.19) 24 1
test_benchmark_query5 0.0067 (1.0) 0.0108 (1.0) 0.0080 (1.0) 0.0009 (1.0) 0.0078 (1.0) 0.0013 (2.12) 25;2 124.4822 (1.0) 98 1
test_benchmark_query6 0.0182 (2.72) 0.0257 (2.39) 0.0212 (2.64) 0.0018 (1.97) 0.0205 (2.65) 0.0029 (4.78) 14;0 47.2173 (0.38) 44 1
test_benchmark_query7 0.1557 (23.32) 0.1673 (15.56) 0.1592 (19.81) 0.0037 (4.10) 0.1581 (20.40) 0.0006 (1.0) 1;2 6.2826 (0.05) 7 1
test_benchmark_query8 3.0889 (462.66) 3.3602 (312.56) 3.2919 (409.78) 0.1153 (126.95) 3.3429 (431.32) 0.1042 (174.43) 1;1 0.3038 (0.00) 5 1
test_benchmark_query9 3.9647 (593.83) 4.0488 (376.61) 4.0125 (499.48) 0.0316 (34.82) 4.0214 (518.87) 0.0398 (66.58) 2;0 0.2492 (0.00) 5 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
=============================================================== 9 passed in 73.77s (0:01:13) ================================================================