为边缘计算领域打造的低时延流式数据处理框架。

wurui b32c6f12e0 fix: return an error if data cannot be unmarshal as FunctionCall (#849) 1 week ago
.github 23aee9230b feat(llm): add Cloudflare OpenAI Gateway support (#764) 3 months ago
ai b32c6f12e0 fix: return an error if data cannot be unmarshal as FunctionCall (#849) 1 week ago
cli 292a148505 feat(llm-sfn): log function calling jsonschema when sfn start (#847) 1 week ago
cmd 6cb2da663d chore: improve comments (#601) 10 months ago
core b32c6f12e0 fix: return an error if data cannot be unmarshal as FunctionCall (#849) 1 week ago
docs f1c84b0a8f chore(deps-dev): bump braces from 3.0.2 to 3.0.3 in /docs (#839) 3 weeks ago
example 92220da02e fix: example (#838) 4 weeks ago
githooks ff4d878c78 feat(core) add rundev 4 years ago
pkg 292a148505 feat(llm-sfn): log function calling jsonschema when sfn start (#847) 1 week ago
rx b5a88b2757 test: enrich unittest & remove dead code (#765) 3 months ago
scripts 9dfb8fb142 refactor: yomo run zipper from a config file (#496) 1 year ago
serverless 273b19a29f refactor: use goroutinue instead of mutex to sync messages (#842) 2 weeks ago
test 5e24c90f96 refacor: change config field downstreams to mesh (#694) 6 months ago
.gitignore 4caef39589 refactor: cli (#802) 2 months ago
.golangci.yml ff4d878c78 feat(core) add rundev 4 years ago
.pre-commit-config.yaml ff4d878c78 feat(core) add rundev 4 years ago
CODE_OF_CONDUCT.md 52e5043a1b Update CODE_OF_CONDUCT.md 4 years ago
CONTRIBUTING.md 296b9b3bba Bump deps (#488) 1 year ago
LICENSE c650727aff chore: Update LICENSE date (#700) 6 months ago
Makefile 23aee9230b feat(llm): add Cloudflare OpenAI Gateway support (#764) 3 months ago
README.md 89608991a0 refactor: serverless context (#822) 1 month ago
Taskfile.yml 0b532139cf refactor: merge CLI repository (#379) 1 year ago
awesome-quic.md 37570a2d3a doc/quic: add learning resources (#39) 3 years ago
build.sh 6f86ef1a3c test: add unit test for ai package (#769) 3 months ago
go.mod 8e0883674b feat(log): 🌈 tinted console logs (#851) 1 week ago
go.sum 8e0883674b feat(log): 🌈 tinted console logs (#851) 1 week ago
main_test.go 9dfb8fb142 refactor: yomo run zipper from a config file (#496) 1 year ago
options.go 273b19a29f refactor: use goroutinue instead of mutex to sync messages (#842) 2 weeks ago
sfn.go 016bdb7380 feat(sfn): llm-sfn read arguments is optional (#846) 2 weeks ago
sfn_test.go b5a88b2757 test: enrich unittest & remove dead code (#765) 3 months ago
source.go 29d215a6b0 feat: client cannot write tag that within reserved range (#815) 1 month ago
source_test.go 29d215a6b0 feat: client cannot write tag that within reserved range (#815) 1 month ago
zipper.go cd899d93ea feat: connector can be customized in NewZipper() (#801) 2 months ago
zipper_notwindows.go b5317dc1a9 refactor: separate the logic of distributed tracing (#736) 4 months ago
zipper_test.go cd899d93ea feat: connector can be customized in NewZipper() (#801) 2 months ago
zipper_windows.go b5317dc1a9 refactor: separate the logic of distributed tracing (#736) 4 months ago

README.md

YoMo Go codecov Discord

YoMo is an open-source LLM Function Calling Framework for building Geo-distributed AI applications. Built atop QUIC Transport Protocol and Stateful Serverless architecture, makes your AI application low-latency, reliable, secure, and easy.

💚 We care about: Customer Experience in the Age of AI

🌶 Features

| | Features | | -- | ------------------------------------------------------------------------------------------------------------ | | ⚡️ | Low-latency Guaranteed by implementing atop QUIC QUIC | | 🔐 | Security TLS v1.3 on every data packet by design | | 📸 | Stateful Serverless Make your GPU serverless 10x faster | | 🌎 | Geo-Distributed Architecture Brings AI inference closer to end users | | 🚀 | Y3 a faster than real-time codec |

🚀 Getting Started

Let's implement a function calling with sfn-currency-converter:

Step 1. Install CLI

curl -fsSL https://get.yomo.run | sh

Verify if the CLI was installed successfully

yomo version

Step 2. Start the server

Prepare the configuration as my-agent.yaml

name: ai-zipper
host: 0.0.0.0
port: 9000

auth:
  type: token
  token: SECRET_TOKEN

bridge:
  ai:
    server:
      addr: 0.0.0.0:8000 ## Restful API endpoint
      provider: openai ## LLM API Service we will use

    providers:
      azopenai:
        api_endpoint: https://<RESOURCE>.openai.azure.com
        deployment_id: <DEPLOYMENT_ID>
        api_key: <API_KEY>
        api_version: <API_VERSION>

      openai:
        api_key: sk-xxxxxxxxxxxxxxxxxxxxxxxxxxx
        model: gpt-4-1106-preview

      gemini:
        api_key: <GEMINI_API_KEY>

      cloudflare_azure:
        endpoint: https://gateway.ai.cloudflare.com/v1/<CF_GATEWAY_ID>/<CF_GATEWAY_NAME>
        api_key: <AZURE_API_KEY>
        resource: <AZURE_OPENAI_RESOURCE>
        deployment_id: <AZURE_OPENAI_DEPLOYMENT_ID>
        api_version: 2023-12-01-preview

Start the server:

YOMO_LOG_LEVEL=debug yomo serve -c my-agent.yaml

Step 3. Write the function

First, let's define what this function do and how's the parameters required, these will be combined to prompt when invoking LLM.

type Parameter struct {
	Domain string `json:"domain" jsonschema:"description=Domain of the website,example=example.com"`
}

func Description() string {
	return `if user asks ip or network latency of a domain, you should return the result of the giving domain. try your best to dissect user expressions to infer the right domain names`
}

func InputSchema() any {
	return &Parameter{}
}

Create a Stateful Serverless Function to get the IP and Latency of a domain:

func Handler(ctx serverless.Context) {
	var msg Parameter
	ctx.ReadLLMArguments(&msg)

	// get ip of the domain
	ips, _ := net.LookupIP(msg.Domain)

	// get ip[0] ping latency
	pinger, _ := ping.NewPinger(ips[0].String())
	pinger.Count = 3
	pinger.Run()
	stats := pinger.Statistics()

	val := fmt.Sprintf("domain %s has ip %s with average latency %s", msg.Domain, ips[0], stats.AvgRtt)
	ctx.WriteLLMResult(val)
}

Finally, let's run it

$ yomo run app.go

time=2024-03-19T21:43:30.583+08:00 level=INFO msg="connected to zipper" component=StreamFunction sfn_id=B0ttNSEKLSgMjXidB11K1 sfn_name=fn-get-ip-from-domain zipper_addr=localhost:9000
time=2024-03-19T21:43:30.584+08:00 level=INFO msg="register ai function success" component=StreamFunction sfn_id=B0ttNSEKLSgMjXidB11K1 sfn_name=fn-get-ip-from-domain zipper_addr=localhost:9000 name=fn-get-ip-from-domain tag=16

Done, let's have a try

$ curl -i -X POST -H "Content-Type: application/json" -d '{"prompt":"compare nike and puma website speed"}' http://127.0.0.1:8000/invoke
HTTP/1.1 200 OK
Content-Length: 944
Connection: keep-alive
Content-Type: application/json
Date: Tue, 19 Mar 2024 13:30:14 GMT
Keep-Alive: timeout=4
Proxy-Connection: keep-alive

{
  "Content": "Based on the data provided for the domains nike.com and puma.com which include IP addresses and average latencies, we can infer the following about their website speeds:
  - Nike.com has an IP address of 13.225.183.84 with an average latency of 65.568333 milliseconds.
  - Puma.com has an IP address of 151.101.194.132 with an average latency of 54.563666 milliseconds.
  
  Comparing these latencies, Puma.com is faster than Nike.com as it has a lower average latency. 
  
  Please be aware, however, that website speed can be influenced by many factors beyond latency, such as server processing time, content size, and delivery networks among others. To get a more comprehensive understanding of website speed, you would need to consider additional metrics and possibly conductreal-time speed tests.",
  "FinishReason": "stop"
}

Full Example Code

Full LLM Function Calling Codes

📚 Documentation

Read more about YoMo at yomo.run/docs.

YoMo ❤️ Vercel, our documentation website is

Vercel Logo

🎯 Focuses on Geo-distributed AI Inference Infra

It’s no secret that today’s users want instant AI inference, every AI application is more powerful when it response quickly. But, currently, when we talk about distribution, it represents distribution in data center. The AI model is far away from their users from all over the world.

If an application can be deployed anywhere close to their end users, solve the problem, this is Geo-distributed System Architecture:

yomo geo-distributed system

🦸 Contributing

First off, thank you for considering making contributions. It's people like you that make YoMo better. There are many ways in which you can participate in the project, for example:

  • File a bug report. Be sure to include information like what version of YoMo you are using, what your operating system is, and steps to recreate the bug.
  • Suggest a new feature.
  • Read our contributing guidelines to learn about what types of contributions we are looking for.
  • We have also adopted a code of conduct that we expect project participants to adhere to.

License

Apache License 2.0