Skip to content

Commit f4567a5

Browse files
pasimoespasimoes
authored
Oracle TEQ Metrics Sample
Oracle TEQ Metrics/Monitoring Sample
1 parent 4679272 commit f4567a5

21 files changed

+2205
-0
lines changed

‎oracle-teq/README.md‎

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
# Monitoring your Oracle Transactional Event Queues
2+
3+
As you may know [Oracle Transactional Event Queues](https://docs.oracle.com/en/database/oracle/oracle-database/21/adque/index.html) is a fault-tolerant, scalable, real-time messaging backbone offered by converged Oracle Database that allows you to build an enterprise-class event-driven architectures.
4+
5+
Have access to the real-time broker, producer, and consumer metrics in a single-pane dashboard and receive alerts for only important issues is key for allows teams to watch and understand the state of their systems.
6+
7+
Oracle Observability Exporter gather the Oracle TEQ metrics and expose through HTTP endpoint as a _target_ to Prometheus collecting and handle them.
8+
9+
## Solution overview
10+
11+
The solution in this session is part of the Unified Observability strategy for Oracle Database but will focus only on Oracle TEQ Metrics handling the two following issues:
12+
13+
- Gather the Oracle TEQ metrics and expose them through an HTTP endpoint using Prometheus standard using Oracle Observability Exporter.
14+
15+
- Deploy Prometheus and Grafana stack to create a single-pane of glass to present Oracle TEQ Metrics on a Grafana Dashboard.
16+
17+
![Solution overview for Metrics](images/unified-observability-metrics.png " ")
18+
19+
## Walkthrough
20+
21+
We provide a method to configure this solution setting up the resources following high-level steps:
22+
23+
1. Create an Oracle TEQ topic.
24+
2. Create Oracle Database secret inside Kubernetes needed for the solution.
25+
3. Create ConfigMap to store TNS_ADMIN data needed to connect with Oracle Database.
26+
4. Create ConfigMap to store Oracle TEQ default metrics configuration file.
27+
5. Deploy Oracle Database Observability Exporter inside Kubernetes.
28+
6. Deploy Service and Monitor Services.
29+
7. Set up Grafana Dashboard to load Oracle TEQ Metrics.
30+
31+
## Prerequisites
32+
33+
For this walkthrough, the following prerequisites are necessary:
34+
35+
- An Oracle Databsase 21c instance running.
36+
- An Oracle User with needed grants to create Oracle TEQ and execute AQ JMS commands.
37+
- A Kubernetes infrastructure running with Grafana and Prometheus instances running and configured.
38+
39+
---
40+
41+
## Set up proactive monitoring in Oracle TEQ
42+
43+
In this section, we walk through the process to set up the resources for active monitoring Oracle TEQ.
44+
45+
### Create an Oracle TEQ topic
46+
47+
We first create a standard Oracle TEQ topic and subscribe to it in order to receive events. We will connect to Oracle Database and execute the script [createTEQ.sql](scripts/createTEQ.sql)
48+
49+
```bash
50+
# Access Oracle Database using SQLcl tool.
51+
sql /nolog
52+
```
53+
54+
```sql
55+
-- Connect using Oracle Database User with needed grants
56+
SQLcl: Release 21.4 Production on Tue Jan 25 00:10:07 2022
57+
58+
Copyright (c) 1982, 2022, Oracle. All rights reserved.
59+
60+
SQL> connect 'DB USER'@'DATABASE HOST:PORT'/'DB SERVICE NAME''
61+
Password? (**********?) ****************
62+
Connected.
63+
```
64+
65+
```sql
66+
--- Run script to create Oracle TEQ Topic.
67+
SQL> @/scripts/createTEQ.sql
68+
SQL>
69+
SQL> begin
70+
2 -- create the TEQ
71+
3 dbms_aqadm.create_transactional_event_queue(
72+
4 queue_name => 'my_teq',
73+
5 -- when mutiple_consumers is true, this will create a pub/sub "topic" - the default is false
74+
6 multiple_consumers => true
75+
7 );
76+
8
77+
9 -- start the TEQ
78+
10 dbms_aqadm.start_queue(
79+
11 queue_name => 'my_teq'
80+
12 );
81+
13 end;
82+
14 /
83+
...
84+
...
85+
...
86+
87+
PL/SQL procedure successfully completed.
88+
```
89+
90+
If you already have an existing TEQ topic that you want to use, you can skip to the next step.
91+
92+
### Create Oracle Database secret
93+
94+
This step creates the K8s secret to store the Oracle Database User credentials that is used by Oracle Observability Exporter.
95+
96+
```bash
97+
kubectl create secret generic db-secret \
98+
--from-literal=username=<"DB USERNAME"> \
99+
--from-literal=password=<"DB PASSWORD"> \
100+
--namespace <"NAMESPACE">
101+
```
102+
103+
> Attention: We assume that Exporter will be deployed in the K8s namespace "NAMESPACE."
104+
105+
### Create ConfigMap to store TNS_ADMIN
106+
107+
This step creates the K8s ConfigMap to store the Oracle Database TNS data that is used by Oracle Observability Exporter.
108+
109+
```bash
110+
kubectl create configmap db-metrics-tns-admin \
111+
--from-file=TNS_ADMIN/tnsnames.ora \
112+
--namespace <"NAMESPACE">
113+
```
114+
115+
### Create ConfigMap to store TEQ Metrics Configuration
116+
117+
This step creates the K8s ConfigMap to store the TEQ metrics configuration file that is used by Oracle Observability Exporter.
118+
119+
```bash
120+
kubectl create configmap db-metrics-teq-exporter-config \
121+
--from-file=metrics/default-metrics-teq.toml \
122+
--namespace <"NAMESPACE">
123+
```
124+
125+
### Deploy Oracle Database Observability Exporter
126+
127+
This step request the deployment of the Oracle Database Observability Exporter in Kubernetes.
128+
129+
```bash
130+
kubectl create -f teq-metrics-exporter-deployment.yaml \
131+
--namespace <"NAMESPACE">
132+
```
133+
134+
```bash
135+
kubectl logs pods/db-metrics-exporter-75948b556f-28btc -n msdataworkshop
136+
137+
. ____ _ __ _ _
138+
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
139+
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
140+
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
141+
' |____| .__|_| |_|_| |_\__, | / / / /
142+
=========|_|==============|___/=/_/_/_/
143+
:: Spring Boot :: (v2.7.0)
144+
145+
2022-07-11 12:07:26.305 INFO 1 --- [ main] o.o.ObservabilityExporterApplication : Starting ObservabilityExporterApplication v0.1.0 using Java 11.0.15 on db-metrics-exporter-75948b556f-28btc with PID 1 (/usr/share/observability-exporter.jar started by root in /)
146+
2022-07-11 12:07:26.313 INFO 1 --- [ main] o.o.ObservabilityExporterApplication : No active profile set, falling back to 1 default profile: "default"
147+
2022-07-11 12:07:28.448 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 9161 (http)
148+
2022-07-11 12:07:28.470 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
149+
2022-07-11 12:07:28.470 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.63]
150+
2022-07-11 12:07:28.611 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
151+
2022-07-11 12:07:28.611 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2174 ms
152+
.......
153+
.......
154+
.......
155+
2022-07-11 12:07:31.856 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 9161 (http) with context path ''
156+
2022-07-11 12:07:31.880 INFO 1 --- [ main] o.o.ObservabilityExporterApplication : Started ObservabilityExporterApplication in 6.697 seconds (JVM running for 7.74)
157+
2022-07-11 12:07:32.370 INFO 1 --- [nio-9161-exec-2] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
158+
2022-07-11 12:07:32.370 INFO 1 --- [nio-9161-exec-2] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
159+
2022-07-11 12:07:32.371 INFO 1 --- [nio-9161-exec-2] o.s.web.servlet.DispatcherServlet : Completed initialization in 1 ms
160+
```
161+
162+
### Deploy Service and Monitor Services
163+
164+
Once we have the `ObservabilityExporterApplication` running, we can expose its service that will be the target monitored by Prometheus ServiceMonitor component, also deployed in this task. First let's deploy the Oracle Observability Exporter service.
165+
166+
```bash
167+
kubectl create -f teq-metrics-exporter-service.yaml \
168+
--namespace <"NAMESPACE">
169+
```
170+
171+
Now let's deploy the Prometheus ServiceMonitor component that will access the Observability Exporter service every 20 seconds and get the metrics made available by it.
172+
173+
```bash
174+
kubectl create -f teq-metrics-exporter-monitor.yaml \
175+
--namespace <"NAMESPACE">
176+
```
177+
178+
### Set up Grafana Dashboard
179+
180+
This step illustrates how to create the Grafana Dashboard to proactively monitoring the Oracle Transactional Event Queues. As an assumption, you already have a Grafana and Prometheus stack up and configured, so we will create the Dashboard to monitor the TEQ.
181+
182+
1. Open a new browser tab in the grafana path as the URL:
183+
184+
`https://<EXTERNAL-IP>/grafana`
185+
186+
![Grafana Home](images/grafana-home.png " ")
187+
188+
2. View pre-configured Prometheus data source:
189+
190+
Select the Configuration gear icon on the left-hand side and select Data Sources.
191+
192+
![grafana-ds-menu](images/grafana-config-datasource-menu.png " ")
193+
194+
Click the Prometheus option.
195+
196+
![Grafana Prometheus Data Source Configuration](images/grafana-config-datasource-prometheus.png " ")
197+
198+
Click Test button and verify success.
199+
200+
![grafana-ds-test](images/grafana-config-datasource-test.png " ")
201+
202+
Click the Back button.
203+
204+
3. Install the Oracle TEQ ("TEQ Monitor") Dashboard
205+
206+
Select the + icon on the left-hand side and select Import
207+
208+
![grafana-dashboard-import](images/grafana-dashboard-import.png " ")
209+
210+
Copy the contents of the [TEQ Dashboard JSON](dashboards/teq-dashboard-basics.json)
211+
212+
Paste the contents in the Import via panel json text field and click the Load button
213+
214+
![grafana-dashboard-import-json](images/grafana-dashboard-import-json.png " ")
215+
216+
Confirm upload and click Import button.
217+
218+
4. Open and Study the TEQ Monitor Dashboard Screen and Metrics
219+
220+
Select the four squares icon on the left-hand side and select 'Dashboards'
221+
222+
In the Dashboards panel query for `TEQ`
223+
224+
![grafana-dashboard-browse](images/grafana-dashboard-browse.png " ")
225+
226+
In the Dashboards panel select `TEQ Monitor` to access Dashboard and study their content which includes metrics about:
227+
- Subscribers
228+
- Message counts, latency, etc.
229+
- Enqueue and Dequeue rates
230+
231+
![grafana-dashboard-teq-02](images/grafana-dashboard-teq-02.png " ")
232+
233+
> Note: the metrics are presented as soon as the events are being produced and consumed in the topics. If you want to simulate you can use the scripts samples [produceTEQ.sql](scripts/produceTEQ.sql) and [consumeTEQL.sql](scripts/consumeTEQ.sql).
234+
235+
## Learn More
236+
237+
- Ask for help and connect with us on the [Oracle DB Microservices Slack Channel](https://bit.ly/oracle-db-microservices-help-slack) channel #oracle-db-microservices
238+
239+
- Experiment our workshop [Unified Observability in Grafana with converged Oracle Database](http://bit.ly/unifiedobservability)
240+
241+
- Our blog [Unified Observability: Metrics, Logs, and Tracing of App and Database Tiers in a Single Grafana Console](https://blogs.oracle.com/developers/post/unified-observability-metrics-logs-and-tracing-of-app-and-database-tiers-in-a-single-grafana-console)

‎oracle-teq/TNS_ADMIN/tnsnames.ora‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
PDB1=
2+
(DESCRIPTION=
3+
(ADDRESS= (PROTOCOL=tcp)(HOST=<DATABASE HOST>)(PORT=<DATABASE PORT>))
4+
(CONNECT_DATA=
5+
(SERVICE_NAME=ORCLPDB1)))

0 commit comments

Comments
 (0)