CHANGELOG.md

Changelog

4.3.0 (2024-06-07)

Features

• Adds timing info to llm_classify (#3377) (3e2785f)
• Serializable execution details (#3358) (fc74513)
• ui: display input and output for tool spans (if available) (#3396) (73312dc)

Bug Fixes

• add separate package installations to notebooks (#3393) (914e3fe)
• filter out undefined (#3383) (e3a2d31)
• percentage sign for alembic configparser (#3403) (87bcd59)

Documentation

• minimum working example with a local llm (#3348) (e4c657c)

4.2.4 (2024-05-28)

Bug Fixes

• update link to openinference spec (#3322) (61dedf8)

Documentation

• Update langchain dependencies in tutorials (#3316) (e403652)

4.2.3 (2024-05-23)

Bug Fixes

• adjust docker tags (#3297) (f097acc)

4.2.2 (2024-05-23)

Bug Fixes

• Tweak release flow (#3295) (8ff19d3)

4.2.1 (2024-05-23)

Bug Fixes

• gql: don't clear data if read_only (#3291) (dbc3203)

4.2.0 (2024-05-23)

Features

• docker image runs as root by default with tags for nonroot and debug images (#3282) (7178c25)

4.1.3 (2024-05-22)

Bug Fixes

• need to check ".get()" because attribute may not be a dict (#3267) (3917fcc)

4.1.2 (2024-05-20)

Bug Fixes

• join on trace_id in get_qa_with_reference (#3248) (a88d4ff)

Documentation

• new updated readme (#3231) (f728447)

4.1.1 (2024-05-17)

Bug Fixes

• resolve rounding issue in postgres (#3232) (3b6c666)

4.1.0 (2024-05-17)

Features

• Add ASGI root path parameter to Phoenix server (#3186) (e27cc5d)

Documentation

• bump base image in kustomize (#3193) (5e8bc3d)
• PHOENIX_WORKING_DIR default value documentation (#3190) (6957bd9)

4.0.3 (2024-05-13)

Bug Fixes

• Always wait a small amount of time between inserts (#3168) (6e18e3c)

4.0.2 (2024-05-11)

Bug Fixes

• Bulk inserter begins first insert immediately (#3151) (7e17cb2)
• unflatten attributes when loading spans from trace_dataset (#3170) (a165023)

4.0.1 (2024-05-09)

Bug Fixes

• coerce input.value to string at ingestion (#3147) (3742ea7)

Documentation

• update kustomize k8s manifests (#3148) (ba166af)

4.0.0 (2024-05-09)

⚠ BREAKING CHANGES

• Remove experimental module (#2945)

Features

• Add log_traces method that sends TraceDataset traces to Phoenix (#2897) (c8f9ed2)
• add a last N time range selector on project / projects pages (#2907) (3c115f8)
• add bedrock claude tracing tutorial (#2919) (b8b5240)
• add default limit to /v1/spans and corresponding client methods (#3026) (e5698d7)
• add gradient start/end to projects table (#2956) (5b6b217)
• add grpc endpoint (#2232) (8bbd136)
• Add indexes on Annotation tables (#3082) (682ecee)
• Add indexes on spans table (#3098) (12d2574)
• add opentelemetry trace instrumentation for Phoenix server (#2990) (6ed494e)
• Add SQL and Code Functionality Eval Templates (#2861) (c7d776a)
• add trace and document evals to GET v1/evaluations (#2910) (79229f2)
• Add user frustration eval (#2928) (406938b)
• Added support for default_headers for azure_openai. (#2917) (6ee5f24)
• convert graphql api to pull trace evaluations from db (#2867) (11aa455)
• Deprecate datasets module, rename to inferences (#2785) (4987ea3)
• experimental: postgres support (a2657d4)
• fetch annotation names (#2964) (6c5d25d)
• fetch document retrieval metrics per span using SQL (#2960) (9fdb765)
• graphql api pulls from db for document evaluations (#2865) (e4b667d)
• grpc interceptor for prometheus (#3056) (610c8fa)
• ingest document evals (#2847) (f3fde50)
• ingest pyarrow span evals into sqlite (#2837) (3a6666c)
• ingest trace annotations (#2852) (792f674)
• make graphql api for span evaluations read from database (#2860) (5adf750)
• move document evaluation summary to pull from db (#2888) (73ca2d7)
• openapi ui for api exploration (#3041) (5b22961)
• persistence: add support for sorting by eval scores and labels (#2977) (44c3068)
• persistence: bulk inserter for spans (#2808) (9ce841e)
• persistence: clear project (#2976) (665c166)
• persistence: clear traces UI (#2988) (a717ff6)
• persistence: dataloader for document retrieval metrics (#2978) (f55c458)
• persistence: dataloader for span descendants (#2980) (d8e10d4)
• persistence: ensure migrations run for TreadSession (#2855) (ec4fea7)
• persistence: fetch latency_ms percentiles using sql with dataloaders (#2818) (48d4643)
• persistence: fetch streaming_last_updated_at (#2819) (d665e49)
• persistence: get or delete projects using sql (#2839) (527b9a9)
• persistence: json binary for postgres (#2849) (29351bf)
• persistence: launch app with persist (#2817) (add6103)
• persistence: make launch_app runnable on tmp directory (#2851) (f41e922)
• persistence: span annotation tables (#2788) (874c61e)
• persistence: span query DSL with SQL (#2911) (7c01420)
• persistence: sql sorting for spans (#2823) (eeafb64)
• persistence: use sqlean v3.45.1 as sqlite engine (#2947) (3b202d7)
• Remove experimental module (#2945) (01758cf)
• restrict project metrics to be last 7 days (#2896) (066bc16)
• span filtering by span evaluations (#2923) (4458ec4)
• Support basic auth (#3061) (3202256)
• support for span evaluations to get evaluations endpoint (#2900) (379e336)
• support pagination on spans resolver (#3046) (2113c5c)
• Update API for OpenAPI compliance (#2866) (0db65d8)
• Update eval summaries to use persistence (#2920) (06eb320)

Bug Fixes

• add the remainder of the sentence (#2903) (64874b8)
• backward compatible truthiness for query from dict parsing (#3124) (b425f9d)
• cartesian product in sql join (#2959) (c96092d)
• cartesian products in get_evaluations (#3081) (64ebec8)
• check payload for legacy project_name (#3125) (d7eae60)
• close delete modal on delete (#3069) (083a467)
• commit insert into alembic_version (#3115) (93a144f)
• disable client-side sorting on trace/span tables (#2958) (139dc3e)
• disable grpc when readonly (#3105) (71ceba9)
• Dockerfile launches Phoenix that listens on IPv6 (#3047) (75cc979)
• eliminate interference on global tracer provider (#2998) (5d7b843)
• Enable listening on IPv6 (#3037) (dee6681)
• ensure recent version of opentelemetry-proto is used (#2948) (33647f5)
• evals: incorrect wording in hallucinations (#3085) (7aa0292)
• fix docker build for sql (b6d508d)
• forbid blank or empty evaluation names (#2962) (cb87977)
• improve error handling and logging for eval insertions (#2854) (d04694b)
• include migration files (#2887) (b0a772e)
• Invalidate cache on project reset (#3113) (2944ae5)
• normalize datetime for phoenix client (#3088) (94a25ae)
• normalize telemetry url before setup (#3001) (28389e8)
• persistence: db race condition between spans and evals (#2905) (2666464)
• persistence: import asert_never from typing_extensions (#2850) (62644cb)
• persistence: postgres down migration and url support (#2915) (4b4a776)
• persistence: postgres json calculations (#2848) (45f084d)
• persistence: postgres timestamp insertion (#2844) (3477bb9)
• preserve loggers across migrations (#2835) (2821bb4)
• prometheus transaction timers for bulkloader (#3066) (e0cc58d)
• Propagate migration errors and show an informative message (#2994) (3718e10)
• remove broken non-asyncio prometheus grpc server interceptor (#3065) (af75151)
• round down time points to facilitate caching (#3079) (42b03c9)
• run docker as nonroot user (#3100) (c640678)
• safely unpack Evaluations proto in bulk inserter (#2869) (50517f7)
• span and trace evaluation summaries (#3013) (088e6c2)
• span event to dict conversion (#3009) (3c73f03)
• switch license format in toml (5c6f345)
• typo in SpanAnnotation (#2967) (f41044e)
• typo in trace annotation table name (#2946) (344b858) Documentation
• Add log_traces tutorial (#2902) (e583f03)
• development: make it explicit that you need to run pnpm build (#3035) (672cbed)
• dockerize manual instrumentation example (#2797) (651efbe)
• manually instrumented chatbot (#2730) (46be32b)
• remove experimental tags in code (4c4a832)

3.25.0 (2024-05-06)

Features

• add bedrock claude tracing tutorial (#2919) (b8b5240)

Bug Fixes

• evals: incorrect wording in hallucinations (#3085) (7aa0292)
• run docker as nonroot user (#3100) (c640678)

Documentation

• development: make it explicit that you need to run pnpm build (#3035) (672cbed)

3.24.0 (2024-04-22)

Features

• Add user frustration eval (#2928) (406938b)

Bug Fixes

• ensure recent version of opentelemetry-proto is used (#2948) (33647f5)

3.23.0 (2024-04-19)

Features

• Added support for default_headers for azure_openai. (#2917) (6ee5f24)

Bug Fixes

• add the remainder of the sentence (#2903) (64874b8)

Documentation

• Add log_traces tutorial (#2902) (e583f03)

3.22.0 (2024-04-16)

Features

• Add log_traces method that sends TraceDataset traces to Phoenix (#2897) (c8f9ed2)

3.21.0 (2024-04-12)

Features

• Add SQL and Code Functionality Eval Templates (#2861) (c7d776a)

3.20.0 (2024-04-10)

Features

• Deprecate datasets module, rename to inferences (#2785) (4987ea3)

Documentation

• dockerize manual instrumentation example (#2797) (651efbe)
• remove experimental tags in code (4c4a832)

3.19.4 (2024-04-04)

Bug Fixes

• switch license format in toml (5c6f345)

Documentation

• fix qa with reference tutorial (e1db1ce)
• fix qa with reference tutorial (ba24950)
• make dockerhub URL go to public (6650f67)
• manually instrumented chatbot (#2730) (46be32b)

3.19.3 (2024-03-30)

Bug Fixes

• ui: show formatted JSON for attributes (0d1b719)
• ui: show formatted JSON for attributes (09ad1be)

3.19.2 (2024-03-29)

Bug Fixes

• ui: broken context for markdown (556e901)

3.19.1 (2024-03-29)

Bug Fixes

• UI: color rotation for markdown (3184359)

3.19.0 (2024-03-29)

Features

• gql: add trace node and trace evaluations (#2662) (a985684)

3.18.1 (2024-03-28)

Bug Fixes

• ignore docs/ directory when formatting (#2714) (1340f74)
• repair frontend build step in release pipeline (#2716) (796eb6a)

3.18.0 (2024-03-28)

Features

• change docker base image to distroless (#2708) (89d6fe7)

3.17.1 (2024-03-24)

Bug Fixes

• long project names do not overflow and squash project icon (#2686) (b77bfaa)

Documentation

• Add mistral (GITBOOK-594) (78676af)
• add mistral instrumentation to notebook (#2681) (54dc47d)
• add mistral instrumentor to mistral tutorial (#2682) (13fc1f8)
• Evals Structure! (GITBOOK-547) (ac23311)
• fix missing parentheses (GITBOOK-571) (2353953)
• Mistral (GITBOOK-595) (f245844)
• No subject (GITBOOK-597) (b6196ac)
• No subject (GITBOOK-598) (f6a2bd6)
• Remove pinecone notebook (#2665) (9f1c1d4)
• trace a deployed app (GITBOOK-593) (08623ea)

3.17.0 (2024-03-21)

Features

• Add response_format argument to MistralAIModel (#2660) (7da51af)
• evals: Add Mistral as an eval model (#2640) (c13ab6b)

Documentation

• example using cron for online phoenix evals (#2643) (5ea99ef)
• mistral tutorial (#2627) (97d4096)

3.16.3 (2024-03-20)

Bug Fixes

• project name for evals (#2648) (14a3c2c)
• trace: query dsl for numpy arrays (#2652) (33f7c73)

3.16.2 (2024-03-20)

Bug Fixes

• trace: redefine root span (#2632) (7940c9d)
• ui: increase pagination size for TracePage (#2642) (6cd456f)

Documentation

• Add Qdrant + Langchain tracing example (#2634) (7f014f8)

3.16.1 (2024-03-19)

Bug Fixes

• trace: eliminate truth ambiguity with non-empty numpy arrays (#2626) (be8ce7d)

Documentation

• Add projects tutorial (#2611) (cca0a0e)

3.16.0 (2024-03-15)

Features

• delete project ui (#2593) (7708805)

3.15.1 (2024-03-15)

Bug Fixes

• handle numpy types in json.dumps for gql (#2600) (13cce4f)

Documentation

• use projects with ragas (#2569) (1e7b31d)

3.15.0 (2024-03-14)

Features

• launch_app() with experimental span storage using environment variables for storage path and storage type enums (#2564) (8a0b572)
• project archiving and deletion (#2585) (121f904)

Bug Fixes

• projects: the home page should direct you to the projects page if there are multiple projects with data (#2586) (ced4e75)
• use environment variable for project name (#2590) (e2ace76)

Documentation

• Improve projects-related API docstrings (#2589) (9eebb00)

3.14.2 (2024-03-14)

Bug Fixes

• increase attributes limit on spans (#2575) (94b1930)
• support numpy arrays in span to json encoder (#2583) (3a297d5)

3.14.1 (2024-03-14)

Bug Fixes

• sanitize base path (#2573) (f2647a2)

3.14.0 (2024-03-14)

Features

• experimental span storage with append-only text files (909672b)
• experimental span storage with append-only text files (#2553) (909672b)

Bug Fixes

• sagemaker: graphql base url was incorrect for sagemaker jupyterlab (#2572) (7ecf46e)

3.13.1 (2024-03-13)

Bug Fixes

• ui: scroll column selector when long (#2552) (cbf8df8)

3.13.0 (2024-03-13)

Features

• add arize-phoenix support for python 3.12 (#2555) (aac0cd5)

3.12.0 (2024-03-13)

Features

• Enable dynamic project switching (#2537) (0ef3224)

Bug Fixes

• prevent browser caching of static assets (#2549) (038e56e)

3.11.1 (2024-03-12)

Bug Fixes

• display newlines in explanations (#2531) (12e8a97)

3.11.0 (2024-03-11)

Features

• graphql: embed project inside graphql span as private attribute (#2522) (9be1afa)
• trace: context manager to pause tracing (#2520) (6bf7232)

Bug Fixes

• parse files to detect sagemaker (#2527) (0761513)

Documentation

• Update pyproject.toml with proper biline (4fdf710)

3.10.0 (2024-03-09)

Features

• projects: add support for the PHOENIX_PROJECT_NAME param (#2515) (6f24786)
• show first non-empty project (#2508) (54a2834)

Bug Fixes

• support minimal llama-index installations (#2516) (2469677)

Documentation

• sync Feb 21, 2024 (#2343) (4e151f3)

3.9.0 (2024-03-08)

Features

• ui: copy to clipboard for prompt template etc. (#2496) (9b853d0)

3.8.0 (2024-03-07)

The Phoenix evals module is graduating out of experimental! You can now install Phoenix evals as a standalone package with pip install arize-phoenix-evals or you can include the new version of phoenix.evals along with the Phoenix install with pip install -U arize-phoenix[evals]. Swapping to the new evals module includes a few small breaking changes which might require some migration work. Details can be found in MIGRATION.md.

phoenix.experimental.evals is being deprecated and will remain in Phoenix for about a month before being removed.

Features

• gql: add trace count to gql project (#2484) (91b4ae1)
• Integrate phoenix.evals into phoenix (#2420) (dd3e7b4)

Documentation

• Add SQL retriever tracing tutorial (#2468) (c92b118)

3.7.0 (2024-03-07)

Features

• projects: project listing (#2459) (2a19814)
• projects: project node interace (#2466) (9d8ade0)

3.6.0 (2024-03-06)

Features

• traces: store and query spans by project name (#2433) (b8ef923)
• ui: auto-expand side nav on hover (#2458) (da83f69)

Bug Fixes

• link to span (#2460) (cbef052)

3.5.0 (2024-03-05)

Features

• add metadata to spans and traces table (#2339) (e9725a2)
• Removes token processing module from phoenix.evals (#2421) (fbd4961)
• ui: new side nav with projects (#2359) (d8c423e)

Bug Fixes

• Properly define BedrockModel (#2425) (81a720c)
• remove computed atributes from exported dataframe (#2366) (1de1415)
• turn span_kind enums into string because it's not serializable by pyarrow (#2438) (50c7eb0)
• update rag and llm ops notebooks (#2442) (adf1b2b)

Documentation

• evals: update tracing tutorials with arize-phoenix-evals (#2386) (1af8187)
• log information about the server at startup (#2445) (6d410c1)
• update readme for phoenix.evals, fix llama-index example (#2435) (dfffaad)

3.4.1 (2024-02-29)

Bug Fixes

• remove symbolic links for docker build (#2408) (b57abe9)
• source distribution build (#2407) (1e67d7e)

3.4.0 (2024-02-28)

Features

• Add phoenix.evals bridge to phoenix and add evals extra install (#2389) (d8b9054)

Bug Fixes

• remove run_relevance_evals and fix import issues (#2375) (9a97e62)
• traces: add y scroll on trace tree (#2399) (9c4f6b9)

Documentation

• evals: add README (#2363) (47842da)
• evals: migrate evaluation notebooks (#2388) (3dedc6e)
• update ragas integration (#2400) (7bebe98)

3.3.0 (2024-02-23)

Features

• display status description under trace info (#2334) (aed925f)
• show span as soon as they arrive (#2353) (88397a5)

Bug Fixes

• use static version in pyproject.toml for packages (#2346) (ef2148c)

Documentation

• update cspell (#2329) (055506f)

3.2.1 (2024-02-16)

Bug Fixes

• evaluate rag notebook (#2316) (4219bf2)
• llama_index_search_and_retrieval_notebook (#2315) (21e5429)

Documentation

• update notebooks for px.Client().log_evaluations() (#2311) (a3ca311)

Miscellaneous Chores

• release 3.2.1 (#2326) (dc2f561)

3.2.0 (2024-02-16)

Features

• px.Client log_evaluations (#2308) (69a4b2b)
• trace: display metadata in the trace page UI (#2304) (fce2d63)

Bug Fixes

• make dspy notebook work on colab (#2306) (a518701)

3.1.2 (2024-02-15)

Bug Fixes

• allow json string for metadata span attribute (#2301) (ec7fbe2)
• ui: safely parse JSON and fallback to string for span attributes (#2293) (e43cdbb)

Documentation

• dspy tutorial notebook (#2288) (f26caaa)

3.1.1 (2024-02-15)

Bug Fixes

• fix: cast message to string in vertexai model (86947a2)

Documentation

• Add bedrock instrumentation notebook (#2285) (6294e36)

3.1.0 (2024-02-15)

Features

• filter spans by metadata values (#2268) (1541b73)

Bug Fixes

• set global session to None if it fails to start (#2286) (6752fd2)
• trace: Make dataset IDs unique by instance for TraceDataset (#2254) (1ac170f)

Documentation

• trace: refactor llama-index tutorials to use 0.10.0 (#2277) (055b8d6)

3.0.3 (2024-02-13)

Bug Fixes

• trace: perform library version compatibility on llama_index (#2272) (89bc510)

3.0.2 (2024-02-13)

Bug Fixes

• run_evals correctly falls back to default responses on error (#2233) (4b2bd39)

3.0.1 (2024-02-09)

Bug Fixes

• handle ndarray during ingestion (#2262) (80114fb)
• working_dir (#2257) (d0f617f)

3.0.0 (2024-02-09)

⚠ BREAKING CHANGES

• replace Phoenix tracers with OpenInference instrumentors (#2190)

Features

• replace Phoenix tracers with OpenInference instrumentors (#2190) (b983c70)

2.11.1 (2024-02-09)

Bug Fixes

• ui: add last_hour, fix end of hour rounding (#2247) (aa4efaf)

2.11.0 (2024-02-08)

Features

• ui: add hour time range (#2244) (2e22518)

Bug Fixes

• evals: properly use kw args for models in notebooks (#2235) (7bd59d5)

2.10.0 (2024-02-07)

Features

• embeddings: add search by text and ID on selection (#2219) (99c480c)

Bug Fixes

• endpoint for client inside ProcessSession (#2211) (82e279e)
• trace: return to /tracing url when dismissing trace slide over (#2222) (ee4ced3)
• traces: warn if collector endpoint is set but launch app is called (#2209) (eb97b8d)

Documentation

• custom instrumentation (GITBOOK-495) (3310ba6)
• update px.Client (GITBOOK-494) (61b427c)

2.9.4 (2024-02-06)

Bug Fixes

• disregard active session if endpoint is provided to px.Client (#2206) (6ec0d23)

2.9.3 (2024-02-05)

Bug Fixes

• absolute path for eval exporter (#2202) (2ac39e9)

2.9.2 (2024-02-05)

Bug Fixes

• localhost address for px.Client (#2200) (e56b66a)

2.9.1 (2024-02-05)

Bug Fixes

• absolute path for urljoin in px.Client (#2199) (ba30a30)

Documentation

• update readme with a deployment guide (#2194) (bf67775)

2.9.0 (2024-02-05)

Features

• phoenix client get_evaluations() and get_trace_dataset() (#2154) (29800e4)
• phoenix client get_spans_dataframe() and query_spans() (#2151) (e44b948)

2.8.0 (2024-02-02)

Features

• Remove model-level tenacity retries (#2176) (66d452c)

Bug Fixes

• broken link and openinference links (#2144) (01fb046)
• databricks check crashes in python console (#2152) (5aeeeff)
• default collector endpoint breaks on windows (#2161) (f1a2007)
• Do not retry when context window has been exceeded (#2126) (ff6df1f)
• remove hyphens from span_id in legacy evaluation fixtures (#2153) (fae859d)

Documentation

• add docker badge (e584ed8)
• Add terminal running steps (GITBOOK-441) (91c6b24)
• No subject (GITBOOK-442) (5c4eb6c)
• No subject (GITBOOK-443) (11f46cb)
• No subject (GITBOOK-444) (fcf2bc9)
• update badge (ddcecea)
• update prompt to reflect rails (GITBOOK-445) (dea6dd6)

Miscellaneous Chores

• change release to 2.8.0 (#2181) (0b7b524)

2.7.0 (2024-01-24)

Features

• persistence: add a PHOENIX_WORKING_DIR env var for setting up a… (#2121) (5fbb2e6)

2.6.0 (2024-01-23)

Features

• add ability to save and load TraceDatasets (#2082) (60c5e5e)
• add get_trace_dataset method to session (#2107) (9754b60)
• evals: Gpt 4 turbo context window size (#2112) (389c1a0)
• launch phoenix with evaluations (#2095) (9656d0c)
• support eval exports for session (#2094) (8757fa8)

Bug Fixes

• Clean up vertex clients after event loop closure (#2102) (202c7ea)
• Determine default async concurrency on a per-model basis (#2096) (b44d8aa)
• Resolves Bedrock model compatibility issues (#2114) (c4a5343)
• show localhost when the notebook is running locally (#2090) (095298d)

Documentation

• evals: update RAG evaluations notebook (#2092) (9ad797a)
• evals: update ragas integration notebook (#2100) (66fb048)

2.5.0 (2024-01-16)

Features

• app: databricks notebook support (#2086) (b517480)

Bug Fixes

• Adjust evaluation templates and rails for Gemini compatibility (#2075) (3a7bfd2)

2.4.1 (2024-01-11)

Bug Fixes

• traces: prevent missing key exception when extracting invocation parameters in llama-index (#2076) (5cc9560)

2.4.0 (2024-01-10)

Features

• add persistence for span evaluations (#2021) (589d482)
• ui: add filter condition snippets (#2049) (567fa54)

Bug Fixes

• Handle missing vertex candidates (#2055) (1d0475a)
• OpenAI clients are not cleaned up after calls to llm_classify (#2068) (3233d56)
• traces: remove nan from log_evaluations (#2056) (df9ed5c)

Documentation

• tracing notebook updates (#2053) (a1e5323)

2.3.0 (2024-01-08)

Features

• evaluator enhancements (#2045) (1cc9c0a)

Bug Fixes

• Remove LiteLLM model support check (#2046) (45d3fe6)

Documentation

• Add demo link, examples getting started (GITBOOK-396) (e987315)
• Add Evaluating Traces Section (GITBOOK-386) (7d72029)
• Add evaluations section for results (GITBOOK-387) (2e74be0)
• Add final thoughts to evaluation (GITBOOK-405) (20eab16)
• add import statement (GITBOOK-408) (23247d7)
• add link (GITBOOK-403) (0be280a)
• eval concepts typo (GITBOOK-394) (7c80d4b)
• eval concepts typos (GITBOOK-393) (62bc99f)
• evaluation concepts typo fix (GITBOOK-390) (2cbc1dc)
• Extract Data from Spans (GITBOOK-383) (440f530)
• fix broken section link (GITBOOK-409) (fee537b)
• fix typos (GITBOOK-391) (c8f5a55)
• fix typos (GITBOOK-402) (3cd973d)
• fix typos (GITBOOK-406) (eaa9bea)
• fix typos (GITBOOK-407) (cad4820)
• Initial draft of evaluation core concept (GITBOOK-385) (67369cf)
• Log Evaluations (GITBOOK-389) (369d79d)
• No subject (GITBOOK-399) (94df884)
• Re-arrange nav (GITBOOK-398) (54a87eb)
• Remove the word golden, simplify title (GITBOOK-395) (a2233b2)
• simplify conceps (GITBOOK-384) (c38f6c2)
• Simplify examples page (GITBOOK-400) (6144158)
• Trace Evaluations Section (GITBOOK-388) (2ffa800)
• Update SECURITY.md (#2029) (363e891)

2.2.1 (2023-12-28)

Bug Fixes

• Do not retry if eval was successful when using SyncExecutor (#2016) (a869190)
• ensure float values are properly encoded by otel tracer (#2024) (b12a894)
• ensure llamaindex spans are correctly encoded (#2023) (3ca6262)
• Use separate versioning file (#2020) (f38eedf)

2.2.0 (2023-12-22)

Features

• Add support for Google's Gemini models via Vertex python sdk (#2008) (caf826c)
• Support first-party Anthropic python SDK (#2004) (a323283)

2.1.0 (2023-12-21)

Features

• instantiate evaluators by criteria (#1983) (9c72616)
• support function calling for run_evals (#1978) (8be325c)
• traces: add v1/traces HTTP endpoint to handle ExportTraceServiceRequest (3c94dea)
• traces: add v1/traces HTTP endpoint to handle ExportTraceServiceRequest (#1968) (3c94dea)
• traces: add retrieval summary to header (#2006) (8af0582)
• traces: evaluation summary on the header (#2000) (965beb0)

Bug Fixes

• make alert icon for exceptions visible (#2001) (e7a6567)

2.0.0 (2023-12-20)

⚠ BREAKING CHANGES

• Update llm_classify and llm_generate interfaces (#1974)

Features

• Add async submission to llm_generate (#1965) (5999133)
• add support for explanations to run_evals (#1975) (5143529)
• evaluation column selectors (#1932) (ed07809)
• openai streaming tool calls (#1936) (6dd14cf)
• support running multiple evals at once (#1742) (79d4473)
• Update llm_classify and llm_generate interfaces (#1974) (9fd35a1)

Bug Fixes

• Add lock failsafe (#1956) (9ddbd9c)
• llama-index extra (#1958) (d9b68eb)
• LlamaIndex compatibility fix (#1940) (052349d)
• Model stability enhancements (#1939) (dca42e0)
• traces: span summary root span filter (#1981) (d286f07)

Documentation

• Add anyscale tutorial (#1941) (e47c8d0)
• autogen link (#1946) (c3fb4ce)
• Clear anyscale tutorial outputs (#1942) (63580a6)
• RAG Evaluation (GITBOOK-378) (429f537)
• sync (#1947) (c72bbac)
• traces: autogen tracing tutorial (#1945) (0fd02ff)
• update rag eval notebook (#1950) (d06b8b7)
• update rag evals docs (#1954) (aa6f36a)
• Using phoenix with HuggingFace LLMs- getting started (#1916) (b446972)

1.9.0 (2023-12-11)

Features

• Add retries to Bedrock (#1927) (2728c3e)

Documentation

• Add LLM Tracing+Evals notebook with keyless example (#1928) (4c4aac6)

1.8.0 (2023-12-10)

Features

• embeddings: audio support (#1920) (61cc550)
• openai streaming function call message support (#1914) (25279ca)

1.7.0 (2023-12-09)

Features

• Instrument LlamaIndex streaming responses (#1901) (f46396e)
• openai async streaming instrumentation (#1900) (06d643b)
• traces: query spans into dataframes (#1910) (6b51435)

Bug Fixes

• traces: span evaluations missing from the header (#1908) (5ace81e)

1.6.0 (2023-12-08)

Features

• openai streaming spans show up in the ui (#1888) (ffa1d41)
• support instrumentation for openai synchronous streaming (#1879) (b6e8c73)
• traces: display document retrieval metrics on trace details (#1902) (0c35229)
• traces: filterable span and document evaluation summaries (#1880) (f90919c)
• traces: graphql query for document evaluation summary (#1874) (8a6a063)

Documentation

• llm ops overview notebook (#1882) (5d15c3c)

1.5.1 (2023-12-06)

Bug Fixes

• Improve rate limiter behavior (#1855) (2530569)

1.5.0 (2023-12-06)

Features

• evals: Human vs AI Evals (#1850) (e96bd27)
• semantic conventions for tool_calls array in OpenAI ChatCompletion messages (#1837) (c079f00)
• support asynchronous chat completions for openai instrumentation (#1849) (f066e10)
• traces: document retrieval metrics based on document evaluation scores (#1826) (3dfb7bd)
• traces: document retrieval metrics on trace / span tables (#1873) (733d233)
• traces: evaluation annotations on traces for associating spans with eval metrics (#1693) (a218a65)
• traces: server-side span filter by evaluation result values (#1858) (6b05f96)
• traces: span evaluation summary (aggregation metrics of scores and labels) (#1846) (5c5c3d6)

Bug Fixes

• allow streaming response to be iterated by user (#1862) (76a2443)
• trace dataset to disc (#1798) (278d344)

Documentation

• RAG evaluation notebook using traces (#1857) (4b67805)
• Retrieval Chunks (GITBOOK-372) (39976d3)

1.4.0 (2023-11-30)

Features

• propagate error status codes to parent spans for improved visibility into trace exceptions (#1824) (1a234e9)

1.3.0 (2023-11-30)

Features

• Add OpenAI Rate limiting (#1805) (115e044)
• evals: show span evaluations in trace details slideout (#1810) (4f0e4dc)
• evaluation ingestion (no user-facing feature is added) (#1764) (7c4039b)
• feature flags context (#1802) (a2732cd)
• Implement asynchronous submission for OpenAI evals (#1754) (30c011d)
• reference link correctness evaluation prompt template (#1771) (bf731df)
• traces: configurable endpoint for the exporter (#1795) (8515763)
• traces: display document evaluations alongside the document (#1823) (2ca3613)
• traces: server-side sort of spans by evaluation result (score or label) (#1812) (d139693)
• traces: show all evaluations in the table" (#1819) (2b27333)
• traces: Trace page header with latency, status, and evaluations (#1831) (1d88efd)

Bug Fixes

• enhance llama-index callback support for exception events (#1814) (8db01df)
• pin llama-index temporarily (#1806) (d6aa76e)
• remove sklearn metrics not available in sagemaker (#1791) (20ab6e5)
• traces: convert (non-list) iterables to lists during protobuf construction due to potential presence of ndarray when reading from parquet files (#1801) (ca72747)
• traces: make column selector sync'd between tabs (#1816) (125431a)

Documentation

• Environment documentation (GITBOOK-370) (dbbb0a7)
• Explanations (GITBOOK-371) (5f33da3)
• No subject (GITBOOK-369) (656b5c0)
• sync for 1.3 (#1833) (4d01e83)
• update default value of variable in run_relevance_eval (GITBOOK-368) (d5bcaf8)

1.2.1 (2023-11-18)

Bug Fixes

• make the app launchable when nest_asyncio is applied (#1783) (f9d5085)
• restore process session (#1781) (34a32c3)

1.2.0 (2023-11-17)

Features

• Add dockerfile (#1761) (4fa8929)
• evals: return partial results when llm function is interrupted (#1755) (1fb0849)
• LiteLLM model support for evals (#1675) (5f2a999)
• sagemaker nobebook support (#1772) (2c0ffbc)

Bug Fixes

• unpin llama-index version in tutorial notebooks (#1766) (5ff74e3)

Documentation

• add instructions for docker build (#1770) (45eb5f2)

1.1.1 (2023-11-16)

Bug Fixes

• update tracer for llama-index 0.9.0 (#1750) (48d0996)

1.1.0 (2023-11-14)

Features

• Evals with explanations (#1699) (2db8141)
• evals: add an output_parser to llm_generate (#1736) (6408dda)

Documentation

• evals: document llm_generate with output parser (#1741) (1e70ec3)

1.0.0 (2023-11-10)

⚠ BREAKING CHANGES

• models: openAI 1.0 (#1716)

Features

• models: openAI 1.0 (#1716) (2564521)

0.1.1 (2023-11-09)

Bug Fixes

• traces: handle AIMessageChunk in langchain tracer by matching prefix in name (#1724) (8654c0a)

0.1.0 (2023-11-08)

Features

• add long-context evaluators, including map reduce and refine patterns (#1710) (0c3b105)
• traces: span table column visibility controls (#1687) (559852f)

Bug Fixes

• add bedrock import (#1695) (dc7f3ef)
• pin openai version below 1.0.0 (#1714) (d21e364)
• traces: Keep traces visible behind the details slideover (#1709) (1c8b8f1)

Documentation

• pin tutorials to openai<1 (#1718) (831c041)