wip

README.md

@@ -0,0 +1,104 @@
+# Apache Prometheus Exporter
+
+Exports Prometheus metrics from Apache access logs.
+
+See the [docker](./docker) folder for an example setup using Docker Compose.
+
+## 1. Configure Apache Access Log Format
+
+The following snippet will create a log format named `prometheus` that includes all information the exporter expects. See [Apache documentation](https://httpd.apache.org/docs/2.4/mod/mod_log_config.html#formats) for explanation of the format.
+
+```apache
+LogFormat "%t %h \"%r\" %>s %O %{ms}T \"%{Referer}i\" \"%{User-Agent}i\"" prometheus
+```
+
+## 2. Configure Apache Virtual Hosts
+
+The following snippet is an example of how you could configure Apache to serve 3 domains from different folders using macros.
+
+Each domain has its own access and error log file. The log files are rotated daily, with a dedicated folder for each day, and a `${APACHE_LOG_DIR}/latest/` folder with hard links to today's log files - this folder will be watched by the exporter.
+
+```apache
+<Macro Logs $domain>
+	ErrorLog "|/usr/bin/rotatelogs -l -f -D -L ${APACHE_LOG_DIR}/latest/$domain.error.log ${APACHE_LOG_DIR}/%Y-%m-%d/$domain.error.log 86400"
+	CustomLog "|/usr/bin/rotatelogs -l -f -D -L ${APACHE_LOG_DIR}/latest/$domain.access.log ${APACHE_LOG_DIR}/%Y-%m-%d/$domain.access.log 86400" prometheus
+</Macro>
+
+<Macro Domain $domain>
+	<VirtualHost *:80>
+		ServerName $domain
+		DocumentRoot /var/www/html/$domain
+		Use Logs $domain
+	</VirtualHost>
+</Macro>
+
+Domain first.example.com
+Domain second.example.com
+Domain third.example.com
+
+UndefMacro Domain
+UndefMacro Logs
+```
+
+In this example, the `first.example.com` domain will be served from `/var/www/html/first.example.com`, and its logs will be written to:
+- `${APACHE_LOG_DIR}/latest/first.example.com.access.log`
+- `${APACHE_LOG_DIR}/latest/first.example.com.error.log`
+
+## 3. Configure the Exporter
+
+The exporter requires the following environment variables:
+
+### `HTTP_HOST`
+
+The host that the HTTP server for metrics will listen on. If omitted, defaults to `127.0.0.1`.
+
+### `ACCESS_LOG_FILE_PATTERN`, `ERROR_LOG_FILE_PATTERN`
+
+The path to the access/error log files. You may use a single wildcard to match multiple files in a folder, or to match multiple folders in one level of the path. Whatever is matched by the wildcard will become the Prometheus label `file`. If there is no wildcard, the `file` label will be empty.
+
+#### Example 1 (File Name Wildcard)
+
+Log files for all domains are in `/var/log/apache2/latest/` and are named `<domain>.access.log` and `<domain>.error.log`. This is the set up from the Apache configuration example above.
+
+**Pattern:** `/var/log/apache2/latest/*.access.log`
+
+- Metrics for `/var/log/apache2/latest/first.example.com.access.log` will be labeled: `first.example.com`
+- Metrics for `/var/log/apache2/latest/first.example.com.error.log` will be labeled: `first.example.com`
+- Metrics for `/var/log/apache2/latest/second.example.com.access.log` will be labeled: `second.example.com`
+
+The wildcard may appear anywhere in the file name.
+
+#### Example 2 (Folder Wildcard)
+
+Every domain has its own folder in `/var/log/apache2/latest/` containing log files named `access.log` and `error.log`.
+
+**Pattern:** `/var/log/apache2/latest/*/access.log`
+
+- Metrics for `/var/log/apache2/latest/first.example.com/access.log` will be labeled: `first.example.com`
+- Metrics for `/var/log/apache2/latest/first.example.com/error.log` will be labeled: `first.example.com`
+- Metrics for `/var/log/apache2/latest/second.example.com/access.log` will be labeled: `second.example.com`
+
+The wildcard must not include any prefix or suffix, so `/*/` is accepted, but `/prefix_*/` or `/*_suffix/` is not.
+
+#### Notes
+
+> At least one access log file and one error log file must be found when the exporter starts, otherwise the exporter immediately exits with an error.
+
+> If a log file is deleted, the exporter will automatically resume watching it if it is re-created later. If you want the exporter to forget about deleted log files, restart the exporter.
+
+## 4. Launch the Exporter
+
+Start the exporter. The standard output will show which log files have been found, the web server host, and the metrics endpoint URL.
+
+Press `Ctrl-C` to stop the exporter.
+
+**Important:** Due to library bugs, the exporter will currently not watch rotated log files. If you want to use this project right now, you will need to add the `-c` flag to `rotatelogs`, and restart the exporter after every rotation.
+
+## 5. Collect Prometheus Metrics
+
+Currently, the exporter exposes only these metrics:
+
+- `apache_requests_total` total number of requests
+- `apache_errors_total` total number of errors
+
+More detailed metrics will be added in the future.

docker/docker-compose.yml

@@ -44,7 +44,8 @@ services:
       - logs:/logs
     environment:
       HTTP_HOST: "0.0.0.0"
-      LOG_FILE_PATTERN: "/logs/*.access.log"
+      ACCESS_LOG_FILE_PATTERN: "/logs/*.access.log"
+      ERROR_LOG_FILE_PATTERN: "/logs/*.error.log"
     restart: "always"
 
 volumes: 

src/apache_metrics.rs

@@ -2,22 +2,26 @@ use prometheus_client::metrics::counter::Counter;
 use prometheus_client::metrics::family::Family;
 use prometheus_client::registry::Registry;
 
+type SingleLabel = (&'static str, String);
+
 #[derive(Clone)]
 pub struct ApacheMetrics {
-	pub requests_total: Family<(&'static str, String), Counter>
+	pub requests_total: Family<SingleLabel, Counter>,
+	pub errors_total: Family<SingleLabel, Counter>
 }
 
 impl ApacheMetrics {
 	pub fn new() -> (Registry, ApacheMetrics) {
 		let mut registry = <Registry>::default();
 		
-		let requests_total = Family::<(&'static str, String), Counter>::default();
-		registry.register("apache_requests", "Number of received requests", Box::new(requests_total.clone()));
-		
 		let metrics = ApacheMetrics {
-			requests_total
+			requests_total: Family::<SingleLabel, Counter>::default(),
+			errors_total: Family::<SingleLabel, Counter>::default()
 		};
 		
+		registry.register("apache_requests", "Number of received requests", Box::new(metrics.requests_total.clone()));
+		registry.register("apache_errors", "Number of logged errors", Box::new(metrics.errors_total.clone()));
+		
 		return (registry, metrics);
 	}
 }

src/log_file_pattern.rs

@@ -6,24 +6,22 @@ use std::path::{Path, PathBuf};
 
 use path_slash::PathExt;
 
-/// Environment variable that determines the path and file name pattern of log files.
+/// Reads and parses an environment variable that determines the path and file name pattern of log files.
 ///
 /// Supports 3 pattern types:
 ///
 /// 1. A simple path to a file.
 /// 2. A path with a wildcard anywhere in the file name.
 /// 3. A path with a standalone wildcard component (i.e. no prefix or suffix in the folder name).
-pub const LOG_FILE_PATTERN: &'static str = "LOG_FILE_PATTERN";
+pub fn parse_log_file_pattern_from_env(variable_name: &str) -> Result<LogFilePattern, String> {
-
+	return match env::var(variable_name) {
-pub fn parse_log_file_pattern_from_env() -> Result<LogFilePattern, String> {
-	return match env::var(LOG_FILE_PATTERN) {
 		Ok(str) => {
-			let pattern_str = Path::new(&str).to_slash().ok_or(format!("Environment variable {} contains an invalid path.", LOG_FILE_PATTERN))?;
+			let pattern_str = Path::new(&str).to_slash().ok_or(format!("Environment variable {} contains an invalid path.", variable_name))?;
 			parse_log_file_pattern_from_str(&pattern_str)
 		}
 		Err(err) => match err {
-			VarError::NotPresent => Err(format!("Environment variable {} must be set.", LOG_FILE_PATTERN)),
+			VarError::NotPresent => Err(format!("Environment variable {} must be set.", variable_name)),
-			VarError::NotUnicode(_) => Err(format!("Environment variable {} contains invalid characters.", LOG_FILE_PATTERN))
+			VarError::NotUnicode(_) => Err(format!("Environment variable {} contains invalid characters.", variable_name))
 		}
 	};
 }

src/log_parser.rs

@@ -0,0 +1,58 @@
+use std::fmt::{Display, Error, Formatter};
+
+pub struct AccessLogLineParts<'a> {
+	pub time: &'a str,
+	pub remote_host: &'a str,
+	pub request: &'a str,
+	pub response_status: &'a str,
+	pub response_bytes: &'a str,
+	pub response_time_ms: &'a str,
+	pub referer: &'a str,
+	pub user_agent: &'a str,
+}
+
+impl Display for AccessLogLineParts<'_> {
+	fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error> {
+		write!(f, "[{}] {} \"{}\" {} {} {} \"{}\" \"{}\"", self.time, self.remote_host, self.request, self.response_status, self.response_bytes, self.response_time_ms, self.referer, self.user_agent)
+	}
+}
+
+impl<'a> AccessLogLineParts<'a> {
+	pub fn parse(line: &'a str) -> Result<AccessLogLineParts<'a>, ParseError> {
+		let (time, line) = extract_between_chars(line, '[', ']').ok_or(ParseError::TimeBracketsNotFound)?;
+		let (remote_host, line) = next_space_delimited_part(line).ok_or(ParseError::RemoteHostNotFound)?;
+		let (request, line) = extract_between_chars(line.trim_start_matches(' '), '"', '"').ok_or(ParseError::RequestNotFound)?;
+		let (response_status, line) = next_space_delimited_part(line).ok_or(ParseError::ResponseStatusNotFound)?;
+		let (response_bytes, line) = next_space_delimited_part(line).ok_or(ParseError::ResponseBytesNotFound)?;
+		let (response_time_ms, line) = next_space_delimited_part(line).ok_or(ParseError::ResponseTimeNotFound)?;
+		let (referer, line) = extract_between_chars(line.trim_start_matches(' '), '"', '"').ok_or(ParseError::RefererNotFound)?;
+		let (user_agent, _) = extract_between_chars(line.trim_start_matches(' '), '"', '"').ok_or(ParseError::UserAgentNotFound)?;
+		Ok(AccessLogLineParts { time, remote_host, request, response_status, response_bytes, response_time_ms, referer, user_agent })
+	}
+}
+
+fn next_space_delimited_part(str: &str) -> Option<(&str, &str)> {
+	return str.trim_start_matches(' ').split_once(' ')
+}
+
+fn extract_between_chars(str: &str, left_side: char, right_side: char) -> Option<(&str, &str)> {
+	let str = str.trim_start_matches(' ');
+	let next_char = str.chars().next()?;
+	return if next_char == left_side {
+		str.get(1..)?.split_once(right_side)
+	} else {
+		None
+	};
+}
+
+#[derive(Debug, Copy, Clone)]
+pub enum ParseError {
+	TimeBracketsNotFound,
+	RemoteHostNotFound,
+	RequestNotFound,
+	ResponseStatusNotFound,
+	ResponseBytesNotFound,
+	ResponseTimeNotFound,
+	RefererNotFound,
+	UserAgentNotFound,
+}

src/log_watcher.rs

@@ -3,47 +3,121 @@ use std::io;
 use std::io::{Error, ErrorKind};
 use std::path::PathBuf;
 
-use linemux::MuxedLines;
+use linemux::{Line, MuxedLines};
 use tokio::sync::mpsc::UnboundedSender;
 
-use crate::ApacheMetrics;
+use crate::{ApacheMetrics, log_parser};
 use crate::log_file_pattern::LogFilePath;
 
-pub async fn read_logs_task(log_files: Vec<LogFilePath>, metrics: ApacheMetrics, shutdown_send: UnboundedSender<()>) {
+#[derive(Copy, Clone, PartialEq)]
-	if let Err(error) = read_logs(log_files, metrics).await {
+enum LogFileKind {
+	Access,
+	Error,
+}
+
+struct LogFileInfo<'a> {
+	pub kind: LogFileKind,
+	pub label: &'a String,
+}
+
+impl<'a> LogFileInfo<'a> {
+	fn get_label_set(&self) -> (&'static str, String) {
+		return ("file", self.label.clone());
+	}
+}
+
+pub async fn watch_logs_task(access_log_files: Vec<LogFilePath>, error_log_files: Vec<LogFilePath>, metrics: ApacheMetrics, shutdown_send: UnboundedSender<()>) {
+	if let Err(error) = watch_logs(access_log_files, error_log_files, metrics).await {
 		println!("[LogWatcher] Error reading logs: {}", error);
 		shutdown_send.send(()).unwrap();
 	}
 }
 
-async fn read_logs(log_files: Vec<LogFilePath>, metrics: ApacheMetrics) -> io::Result<()> {
+struct LogWatcher<'a> {
-	let mut file_reader = MuxedLines::new()?;
+	reader: MuxedLines,
-	let mut label_lookup: HashMap<PathBuf, &String> = HashMap::new();
+	files: HashMap<PathBuf, LogFileInfo<'a>>,
-	
+}
-	for log_file in &log_files {
+
-		let lookup_key = file_reader.add_file(&log_file.path).await?;
+impl<'a> LogWatcher<'a> {
-		label_lookup.insert(lookup_key, &log_file.label);
+	fn new() -> io::Result<LogWatcher<'a>> {
+		return Ok(LogWatcher {
+			reader: MuxedLines::new()?,
+			files: HashMap::new(),
+		});
 	}
 	
-	if log_files.is_empty() {
+	fn count_files_of_kind(&self, kind: LogFileKind) -> usize {
-		println!("[LogWatcher] No log files provided.");
+		return self.files.values().filter(|info| info.kind == kind).count();
-		return Err(Error::from(ErrorKind::Unsupported));
 	}
 	
-	println!("[LogWatcher] Watching {} log file(s).", log_files.len());
+	async fn add_file(&mut self, log_file: &'a LogFilePath, kind: LogFileKind) -> io::Result<()> {
+		let lookup_key = self.reader.add_file(&log_file.path).await?;
+		self.files.insert(lookup_key, LogFileInfo { kind, label: &log_file.label });
+		Ok(())
+	}
 	
-	loop {
+	async fn start_watching(&mut self, metrics: &ApacheMetrics) -> io::Result<()> {
-		let event_result = file_reader.next_line().await?;
+		if self.files.is_empty() {
-		if let Some(event) = event_result {
+			println!("[LogWatcher] No log files provided.");
-			match label_lookup.get(event.source()) {
+			return Err(Error::from(ErrorKind::Unsupported));
-				Some(&label) => {
+		}
-					println!("[LogWatcher] Received line from \"{}\": {}", label, event.line());
+		
-					metrics.requests_total.get_or_create(&("file", label.clone())).inc();
+		println!("[LogWatcher] Watching {} access log file(s) and {} error log file(s).", self.count_files_of_kind(LogFileKind::Access), self.count_files_of_kind(LogFileKind::Error));
-				}
+		
-				None => {
+		for metadata in self.files.values() {
-					println!("[LogWatcher] Received line from unknown file: {}", event.source().display());
+			let label_set = metadata.get_label_set();
-				}
+			let _ = metrics.requests_total.get_or_create(&label_set);
+			let _ = metrics.errors_total.get_or_create(&label_set);
+		}
+		
+		loop {
+			if let Some(event) = self.reader.next_line().await? {
+				self.handle_line(event, metrics);
 			}
 		}
 	}
+	
+	fn handle_line(&self, event: Line, metrics: &ApacheMetrics) {
+		if let Some(file) = self.files.get(event.source()) {
+			match file.kind {
+				LogFileKind::Access => self.handle_access_log_line(event.line(), file, metrics),
+				LogFileKind::Error => self.handle_error_log_line(event.line(), file, metrics),
+			}
+		} else {
+			println!("[LogWatcher] Received line from unknown file: {}", event.source().display());
+		}
+	}
+	
+	fn handle_access_log_line(&self, line: &str, file: &LogFileInfo, metrics: &ApacheMetrics) {
+		match log_parser::AccessLogLineParts::parse(line) {
+			Ok(parts) => {
+				println!("[LogWatcher] Received access log line from \"{}\": {}", file.label, parts)
+			}
+			Err(err) => {
+				println!("[LogWatcher] Received access log line from \"{}\" with invalid format ({:?}): {}", file.label, err, line)
+			}
+		}
+		
+		metrics.requests_total.get_or_create(&file.get_label_set()).inc();
+	}
+	
+	fn handle_error_log_line(&self, line: &str, file: &LogFileInfo, metrics: &ApacheMetrics) {
+		println!("[LogWatcher] Received error log line from \"{}\": {}", file.label, line);
+		metrics.errors_total.get_or_create(&file.get_label_set()).inc();
+	}
+}
+
+async fn watch_logs(access_log_files: Vec<LogFilePath>, error_log_files: Vec<LogFilePath>, metrics: ApacheMetrics) -> io::Result<()> {
+	let mut watcher = LogWatcher::new()?;
+	
+	for log_file in &access_log_files {
+		watcher.add_file(log_file, LogFileKind::Access).await?;
+	}
+	
+	for log_file in &error_log_files {
+		watcher.add_file(log_file, LogFileKind::Error).await?;
+	}
+	
+	watcher.start_watching(&metrics).await?;
+	Ok(())
 }

src/main.rs

@@ -1,54 +1,73 @@
 use std::env;
+use std::process::ExitCode;
 use std::sync::Mutex;
 
 use tokio::signal;
 use tokio::sync::mpsc;
 
 use crate::apache_metrics::ApacheMetrics;
-use crate::log_file_pattern::parse_log_file_pattern_from_env;
+use crate::log_file_pattern::{LogFilePath, parse_log_file_pattern_from_env};
-use crate::log_watcher::read_logs_task;
+use crate::log_watcher::watch_logs_task;
 use crate::web_server::{create_web_server, run_web_server};
 
-mod log_file_pattern;
-mod log_watcher;
 mod apache_metrics;
+mod log_file_pattern;
+mod log_parser;
+mod log_watcher;
 mod web_server;
 
-#[tokio::main(flavor = "current_thread")]
+const ACCESS_LOG_FILE_PATTERN: &'static str = "ACCESS_LOG_FILE_PATTERN";
-async fn main() {
+const ERROR_LOG_FILE_PATTERN: &'static str = "ERROR_LOG_FILE_PATTERN";
-	let host = env::var("HTTP_HOST").unwrap_or(String::from("127.0.0.1"));
+
-	
+fn find_log_files(environment_variable_name: &str, log_kind: &str) -> Option<Vec<LogFilePath>> {
-	println!("Initializing exporter...");
+	let log_file_pattern = match parse_log_file_pattern_from_env(environment_variable_name) {
-	
-	let log_file_pattern = match parse_log_file_pattern_from_env() {
 		Ok(pattern) => pattern,
 		Err(error) => {
 			println!("Error: {}", error);
-			return;
+			return None;
 		}
 	};
 	
 	let log_files = match log_file_pattern.search() {
 		Ok(files) => files,
 		Err(error) => {
-			println!("Error searching log files: {}", error);
+			println!("Error searching {} files: {}", log_kind, error);
-			return;
+			return None;
 		}
 	};
 	
 	if log_files.is_empty() {
-		println!("Found no matching log files.");
+		println!("Found no matching {} files.", log_kind);
-		return;
+		return None;
 	}
 	
 	for log_file in &log_files {
-		println!("Found log file: {} (label \"{}\")", log_file.path.display(), log_file.label);
+		println!("Found {} file: {} (label \"{}\")", log_kind, log_file.path.display(), log_file.label);
 	}
 	
+	return Some(log_files);
+}
+
+#[tokio::main(flavor = "current_thread")]
+async fn main() -> ExitCode {
+	let host = env::var("HTTP_HOST").unwrap_or(String::from("127.0.0.1"));
+	
+	println!("Initializing exporter...");
+	
+	let access_log_files = match find_log_files(ACCESS_LOG_FILE_PATTERN, "access log") {
+		Some(files) => files,
+		None => return ExitCode::FAILURE,
+	};
+	
+	let error_log_files = match find_log_files(ERROR_LOG_FILE_PATTERN, "error log") {
+		Some(files) => files,
+		None => return ExitCode::FAILURE,
+	};
+	
 	let (metrics_registry, metrics) = ApacheMetrics::new();
 	let (shutdown_send, mut shutdown_recv) = mpsc::unbounded_channel();
 	
-	tokio::spawn(read_logs_task(log_files, metrics.clone(), shutdown_send.clone()));
+	tokio::spawn(watch_logs_task(access_log_files, error_log_files, metrics.clone(), shutdown_send.clone()));
 	tokio::spawn(run_web_server(create_web_server(host.as_str(), 9240, Mutex::new(metrics_registry))));
 	
 	drop(shutdown_send);
@@ -62,4 +81,6 @@ async fn main() {
 			println!("Shutting down...");
 		}
 	}
+	
+	ExitCode::SUCCESS
 }

src/web_server.rs

@@ -23,7 +23,7 @@ pub fn create_web_server(host: &str, port: u16, metrics_registry: Mutex<Registry
 	let server = server.workers(1);
 	let server = server.bind((host, port));
 
-	println!("[WebServer] Starting web server on http://{}:{}", host, port);
+	println!("[WebServer] Starting web server on {0}:{1} with metrics endpoint: http://{0}:{1}/metrics", host, port);
 	return server.unwrap().run();
 }