{ "model_details": { "provider": "google", "model_requested": "gemma-4-26b-a4b-it", "model_for_requests": "gemma-4-26b-a4b-it", "api_base_url": "https://generativelanguage.googleapis.com/v1beta/openai", "chat_completions_endpoint": "https://generativelanguage.googleapis.com/v1beta/openai/chat/completions" }, "run_config": { "input": [ "data/input/like_interrater.csv" ], "labels": "", "task_name": "*like* discourse/pragm", "task_description": "Classification of instance of *like* into four discourse-pragmatic categories: 1. Quotative marker, 2. Discourse marker, 3. Clause-medial pragmatic marker, 4 . Clause-final pragmatic marker", "tags": "open source; *like*; discourse; pragmatics; English; v3", "model": "gemma-4-26b-a4b-it", "temperature": null, "top_p": null, "top_k": null, "service_tier": "standard", "verbosity": null, "reasoning_effort": null, "thinking_level": null, "effort": null, "strict_control_acceptance": true, "provider": "google", "system_prompt": "You are a meticulous linguistic classifier. Classify each instance of _\\*like\\*_ into \\*\\*one\\*\\* of the four discourse-pragmatic categories:\n1. Quotative marker \n2. Discourse marker \n3. Clause-medial pragmatic marker \n4. Clause-final pragmatic marker\n \nFollow the order of classification precisely as specified: \n**Step 1 - Test Omissibility**\nDetermine whether like is omissible without affecting the propositional meaning or grammatical well-formedness of the clause.\n* If not omissible, first check for quotative marker use.\n**Quotative marker** \nClassify as Quotative marker if like occurs in a quotative frame such as:\n* be like, it’s like, verbs of saying or thinking + like (_say, go, think_)\n* like before reported material and introduces:\n* reported speech, or\n* thoughts, feelings, attitudes, or inner monologue.\n\nExamples: \\* She was like do not tempt me! \\* \\*And I’m like: „I cannot tell them what I am thinking.“ \\* \\* And everyone was like damn. \\* If not, continue. \n\n**Step 2: Check non-quotative** _**it’s like**\n_If _like_ occurs in the construction _it’s like_, is not followed by quoted material, and the whole construction can be omitted without affecting grammaticality or propositional meaning, classify it as a Discourse marker.\nThis construction may function as:\n* discourse link (elaboration, clarification, exemplification)\n* topic-orientation marker\n* disfluency marker\n \nExample: \\*He‘s just fired me. It’s like I don’t understand why. \\* \\*It’s like erm uh, I wanted to see you. \\* If _like_ is not part of quotative construction, or discourse marker _it’s like_, and it cannot be omitted without affecting grammaticality and propositional content of its host unit, the use is propositional (verb, noun, preposition, adverb) and should be classified as \"unclassified\". \n\n**Step 3 — If** _**like**_ **is omissible, classify it primarily by position, and use the functional descriptions and examples only to confirm that the positional classification is appropriate.** \n**Discourse marker**\nIf _like_ occurs clause-initially or outside clause structure, classify as Discourse marker.\nFunction: organises discourse. \nSubfunctions:\n* discourse link (elaboration, clarification, exemplification)\n* topic orientation (topic shift, digression, topic return)\n* disfluency marker (false start, repair, repetition, pause filler)\n\nExamples: \\*It was really late and like we just left. \\*; \\*I don’t know what happened erm like uh I don’t remember. \\* If not, continue. \n\n**Clause-medial pragmatic marker**\nIf _like_ occurs clause-medially, inside the clause before a word or phrase, classify as Clause-medial pragmatic marker.\nFunction: scopes forward over following material.\nSubfunctions:\n* marker of non-equivalence between what is said and what was meant:\n* requiring loosening (figurative/non-literal interpretation)\n* requiring enrichment (vague or underspecified expression)\n* focus marker\n* highlights salient information, including intensification or exemplification\n \nExamples: \\*That’s like really weird.\\* \\*They only have like bread and onions. \\*\\* I like almost died. \\* \\*It was this like big round thing. \\* If not, continue. \n**Clause-final pragmatic marker**\nIf _like_ occurs at the end of a clause or utterance, classify as Clause-final pragmatic marker.\nFunction: scopes backward over preceding material.\nSubfunctions:\n* focus marker (clearing up misunderstandings, emphasis)\n* marker of non-equivalence\n* invariant tag\n* checking understanding\n* requesting confirmation\n \nExample: \\*He’s a bastard, like. \\* \\* I simply cleaned it with vinegar, like. \\* \\*But I think he already paid, like? \\*", "system_prompt_b64": null, "few_shot_examples": 0, "prompt_layout": "standard", "cache_pad_target_tokens": 0, "prompt_cache_key": null, "gemini_cached_content": null, "requesty_auto_cache": null, "vertex_auto_adc_login": null, "vertex_access_token_refresh_seconds": null, "create_gemini_cache": false, "gemini_cache_ttl": 3600, "gemini_cache_ttl_autoupdate": true, "keep_gemini_cache": false, "enable_cot": true, "no_explanation": false, "logprobs": false, "calibration": true, "confusion_heatmap": true, "api_key_var": "GOOGLE_API_KEY", "api_base_var": "GOOGLE_BASE_URL", "max_retries": 3, "retry_delay": 5.0, "request_interval_ms": 0, "request_timeout_seconds": 600.0, "threads": 3, "prompt_log_detail": "full", "flush_rows": 100, "flush_seconds": 2.0, "validator_cmd": null, "validator_args": "", "validator_timeout": 5.0, "validator_prompt_max_candidates": 50, "validator_prompt_max_chars": 8000, "validator_exhausted_policy": "accept_blank_confidence", "validator_debug": false, "log_level": "INFO" }, "source_input_csv": "data/input/like_interrater.csv", "source_output_csv": "data/output/like_interrater__google__modelsgemma426ba4bit__2026-04-04-01-49.csv", "source_labels_csv": "", "cache_padding": { "enabled": false, "target_shared_prefix_tokens": 0, "calibration_shared_prefix_tokens": null, "target_prompt_tokens": 0, "calibration_prompt_tokens": null, "calibration_example_id": null, "applied_padding_tokens_estimate": 0, "examples_with_padding_applied": 0 }, "request_control_summary": { "configured": {}, "attempts_total": 144, "attempts_with_control_telemetry": 144, "per_control": { "reasoning_effort": { "configured_value": null, "requested_attempts": 0, "sent_attempts": 0, "accepted_attempts": 0, "rejected_attempts": 0, "missing_from_final_request_attempts": 0, "acceptance_rate": null, "rejected_reasons": {}, "rejected_example_ids": [] }, "thinking_level": { "configured_value": null, "requested_attempts": 0, "sent_attempts": 0, "accepted_attempts": 0, "rejected_attempts": 0, "missing_from_final_request_attempts": 0, "acceptance_rate": null, "rejected_reasons": {}, "rejected_example_ids": [] }, "effort": { "configured_value": null, "requested_attempts": 0, "sent_attempts": 0, "accepted_attempts": 0, "rejected_attempts": 0, "missing_from_final_request_attempts": 0, "acceptance_rate": null, "rejected_reasons": {}, "rejected_example_ids": [] }, "verbosity": { "configured_value": null, "requested_attempts": 0, "sent_attempts": 0, "accepted_attempts": 0, "rejected_attempts": 0, "missing_from_final_request_attempts": 0, "acceptance_rate": null, "rejected_reasons": {}, "rejected_example_ids": [] }, "prompt_cache_key": { "configured_value": null, "requested_attempts": 0, "sent_attempts": 0, "accepted_attempts": 0, "rejected_attempts": 0, "missing_from_final_request_attempts": 0, "acceptance_rate": null, "rejected_reasons": {}, "rejected_example_ids": [] }, "gemini_cached_content": { "configured_value": null, "requested_attempts": 0, "sent_attempts": 0, "accepted_attempts": 0, "rejected_attempts": 0, "missing_from_final_request_attempts": 0, "acceptance_rate": null, "rejected_reasons": {}, "rejected_example_ids": [] }, "requesty_auto_cache": { "configured_value": null, "requested_attempts": 0, "sent_attempts": 0, "accepted_attempts": 0, "rejected_attempts": 0, "missing_from_final_request_attempts": 0, "acceptance_rate": null, "rejected_reasons": {}, "rejected_example_ids": [] } } }, "usage_metadata_summary": { "attempts_total": 144, "attempts_with_usage_metadata": 144, "attempts_with_cached_token_signals": 0, "cached_tokens_total_estimate": 0, "cache_read_tokens_total": 0, "cache_write_tokens_total": 0, "cache_token_fields_totals": {}, "attempts_with_gemini_cached_content_token_signals": 0, "gemini_cached_content_token_count_total": 0, "gemini_cached_content_token_fields_totals": {} }, "token_usage_totals": { "attempts_total": 144, "attempts_with_token_usage": 144, "attempts_with_output_tokens": 144, "attempts_with_cached_input_tokens": 0, "attempts_with_thinking_tokens": 0, "input_tokens_total": 204628, "cached_input_tokens_total": 0, "non_cached_input_tokens_total": 204628, "output_tokens_total": 273248, "thinking_tokens_total": 0, "output_tokens_definition": "total_tokens - prompt_tokens (or completion_tokens + thinking_tokens fallback)" }, "truth_label_count": 115, "prediction_count": 115, "evaluated_example_count": 115, "calibration_metrics": { "available": true, "sample_count": 114, "bin_count": 10, "ece": 0.13947368421052705, "mce": 0.85, "brier_score": 0.15399122807017543 }, "first_prompt_timestamp": "2026-04-03T23:49:56.734262Z", "last_prompt_timestamp": "2026-04-04T00:31:04.686009Z", "overall_time_seconds": 2467.951747, "overall_time_human": "41m 08s", "accuracy": 0.8173913043478261, "cohen_kappa": 0.7577733199598797, "macro_precision": 0.8220833333333333, "macro_recall": 0.825, "macro_f1": 0.8137628161312372, "per_label": { "Clause-final pragmatic marker": { "precision": 1.0, "recall": 0.68, "f1": 0.8095238095238095, "support": 25 }, "Clause-medial pragmatic marker": { "precision": 0.71875, "recall": 0.92, "f1": 0.8070175438596492, "support": 25 }, "Discourse marker": { "precision": 0.775, "recall": 0.775, "f1": 0.775, "support": 40 }, "Quotative marker": { "precision": 0.95, "recall": 0.95, "f1": 0.9500000000000001, "support": 20 }, "unclassified": { "precision": 0.6666666666666666, "recall": 0.8, "f1": 0.7272727272727272, "support": 5 } }, "labels": [ "Clause-final pragmatic marker", "Clause-medial pragmatic marker", "Discourse marker", "Quotative marker", "unclassified" ], "label_count": 5, "total_examples": 115, "confusion_matrix": { "Clause-final pragmatic marker": { "Clause-final pragmatic marker": 17, "Clause-medial pragmatic marker": 3, "Discourse marker": 5, "Quotative marker": 0, "unclassified": 0 }, "Clause-medial pragmatic marker": { "Clause-final pragmatic marker": 0, "Clause-medial pragmatic marker": 23, "Discourse marker": 2, "Quotative marker": 0, "unclassified": 0 }, "Discourse marker": { "Clause-final pragmatic marker": 0, "Clause-medial pragmatic marker": 6, "Discourse marker": 31, "Quotative marker": 1, "unclassified": 2 }, "Quotative marker": { "Clause-final pragmatic marker": 0, "Clause-medial pragmatic marker": 0, "Discourse marker": 1, "Quotative marker": 19, "unclassified": 0 }, "unclassified": { "Clause-final pragmatic marker": 0, "Clause-medial pragmatic marker": 0, "Discourse marker": 1, "Quotative marker": 0, "unclassified": 4 } }, "label_metrics_available": true }